What is the resource_semaphore wait?

A wait type of resource_semaphore means that there isn’t enough available memory to grant for queries to run. At a high level, here is what is happening. A query is submitted to the SQL Server engine for execution. As part of the pre-execution phase, SQL Server estimates how much memory it thinks a query will need to run. Several factors influence this memory estimation. Assuming there is free memory to grant to the query, then the query moves along the execution phases and starts running.

But let’s say you have a server with 128 GB of RAM allocated to SQL Server. A series of queries are submitted to the SQL Server database engine that are each granted 15 GB of RAM. At most, SQL Server can handle 8 of those queries before it runs out of memory to allocate. The next query that comes along and needs another 15 GB of RAM is prevented from starting its execution because 8 X 15 = 120 GB of RAM. This 9th query, if granted memory, would cause a total of 135 GB to be allocated. The server doesn’t have that much RAM allocated for queries. So, it has to wait.

As other queries are submitted, they wait behind this 9th query that needs the additional 15 GB of RAM. If the other 8 queries that are already executing are long-running queries, it might be several minutes, or longer, before memory is available. Queries start stacking up behind each other. Users start noticing that the app is slow, pages aren’t refreshing, and reports aren’t completing. Soon, everyone is hitting F5 on the web app to resubmit queries because nothing is happening. Immediately following this, your phone or your Slack messages start blowing up!

How to find queries with large memory grants

First, you can use sp_whoisactive to find the queries and the offending wait in real-time. This is very useful for the scenario above, where users feel the pain and start entering tickets and messaging people for help.

However, let’s say, for example, that your server is just teetering on the edge with the memory allocation. Memory grants are occasionally high enough that SQL Server is registering the resource_semaphore wait, but it isn’t happening for long enough, or frequently enough, that users really notice and start complaining. Your SQL Server is experiencing slowness, at times, it’s just not causing excruciating pain. This may show up by this wait appearing low in the result set from Paul Randall’s wait stats query. Maybe it’s only causing a few seconds of wait, on average, when it happens. Users might notice this because their report or application screen completes after a brief wait, so they assume this is “normal.” Consequently, they don’t report it. This doesn’t mean you can or should ignore what the wait stats information is telling you. Act now before this becomes a full-blown emergency.

Second, in the above scenario, you can use Extended Events to find queries with large memory grants. Extended Events are light-weight trace objects that allow for the capture of far more events than Profiler or a server-side trace. They also work differently by only firing and capturing when an event defined in the session happens, versus capturing everything and then filtering like Profiler does.

Setting Up The Extended Events Session

After some poking around and some experimenting, I was able to arrive at the T-SQL below to create the extended event session. This code will capture any query with a memory grant greater than 1 GB. Adjust this higher or lower as makes sense for your environment. The session stores the database ID, plan handle, session ID, and the T-SQL text for the offending query in a file. It will write to as many as 5 files, each 1 GB in size. When the 5th file is full, it will delete the oldest files and start writing a new file.

One thing to be aware of is the path for the file that will hold the data. That path must exist first. I’m using C:\XE\NameofExtendedEventSession.xel. Be sure to update that path to a location that your SQL Server instance can access.

CREATE EVENT SESSION [TrackHighMemoryGrants] ON SERVER 
ADD EVENT sqlserver.query_memory_grant_usage(
ACTION(sqlserver.database_id,sqlserver.plan_handle,sqlserver.session_id,sqlserver.sql_text)
    WHERE ([granted_memory_kb]>(1024000))),
ADD EVENT sqlserver.query_memory_grant_wait_end(
    ACTION(sqlserver.database_id,sqlserver.session_id,sqlserver.sql_text)
    WHERE ([sqlserver].[database_id]=(8) AND [granted_memory_kb]>(1024000)))
ADD TARGET package0.event_file(SET filename=N'C:\XE\HighMemoryGrants.xel',max_file_size=(1024),max_rollover_files=(5))
WITH (MAX_MEMORY=51200 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=5 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=ON,STARTUP_STATE=OFF)
GO

To start the Extended Events session, run the following T-SQL:

ALTER EVENT SESSION [TrackingHighMemoryGrants]
ON SERVER
STATE = START;

This can also be done from SSMS. Traverse the UI under the SQL instance name to Management > Extended Events > “TrackingHighMemoryGrants”, right click and select the “Start Session” option.

How to query Extended Events files

The extended event is gathering data. The query below can parse the collected files to show which queries have the highest memory grants on average.

WITH ParsedEvents AS (
SELECT 
event_data.value('(event/action[@name="sql_text"]/value)[1]', 'nvarchar(max)') AS sql_text,
event_data.value('(event/data[@name="granted_memory_kb"]/value)[1]', 'bigint') / 1024.0 AS granted_mb
FROM (
    SELECT CAST(event_data AS XML) AS event_data
    FROM sys.fn_xe_file_target_read_file('C:\XE\HighMemoryGrants*.xel', NULL, NULL, NULL)
    ) AS x
)

SELECT 
sql_text,
COUNT(*) AS execution_count,
AVG(granted_mb) AS avg_granted_mb,
MAX(granted_mb) AS max_granted_mb,
MIN(granted_mb) AS min_granted_mb
FROM ParsedEvents
GROUP BY sql_text
ORDER BY avg_granted_mb DESC;

Now you can start dealing with the queries involved in that pesky wait before the problem brings your server to a screeching halt! By the way, the queries may lead you to a design problem in your tables that can cause high memory grants. More about that in a future post!

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post How to Find Queries Causing RESOURCE_SEMAPHORE Waits in SQL Server appeared first on The SERO Group.

A Refresher on the SQL Server Maturity Curve

First, let’s remind ourselves of what the progression of the SQL Server maturity curve looks like, as discussed in this blog.

1. Reactive—Firefighting
2. Managed—Gaining control
3. Optimized—Running proactively
4. Strategic—Turning data into advantage

How SQL Server Query Store Helps Businesses Take Control

Query Store’s features can help shift businesses away from break/fix firefighting. Because the Query Store collects runtime metrics for query performance, as well as query plans, businesses aren’t running blind anymore.

You want to know what was going on last Tuesday at 3:15PM when the app was slow? Look at Query Store to see what queries were running and what plans the queries were using. There may have been a change in the query plan, and after that is when people reported that the app was slow. Those query plans may also reveal missing indexes that can help performance.

Additionally, the Query Store report called “Top Resource Consuming Queries” will show how much CPU and RAM were involved in those queries. The Query Wait Statistics report will represent the resources that queries were waiting on as very visible bar graphs. This report enables IT professionals to see that issues like locks or disk IO are affecting their queries.

This type of data gives you the insight needed to start gaining control over your environment.

Moving from Managed to Strategic with Query Store

Now your business can move to the next stage of maturity, which is being proactive. For example, now you can start rewriting queries that were running when the application was slow.  Those missing index recommendations found in various query plans can be evaluated and compared to existing indexes. Perhaps a recommended and existing index can be combined into a single index that can be tested in a lower environment to see if it has a positive impact on the query plan, resulting in lower query durations or lower resource usage.

If the application is from a vendor, then you likely can’t make changes to the queries or indexing, but you now have data to provide the vendor when approaching them with performance problems.

Additionally, you can do things like forcing certain query plans, and in SQL Server 2022, you can apply query hints within Query Store so that even if an application is from a 3^rd party vendor, you can still make improvements without changing the application code. Query Hints are stored within the Query Store and applied after the code is submitted to SQL Server.

Query Store, then, empowers your business to be strategic. The team can see where the pain points are, test and prioritize issues and solutions, and hand over data to vendors when they can’t make changes themselves. This proactive, strategic work can literally save your business a lot of money by lowering resource usage, allowing for a smaller SQL Server instance. This saves money on licensing and, if you are in the cloud, saves money spent on the size of resources provisioned.

Want to Work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post How Query Store Drives Progress Along the SQL Server Maturity Curve appeared first on The SERO Group.

One thing to point out is that in SQL Server 2022 and above, when creating a new database from the SSMS GUI or by simply using the CREATE DATABASE MyNewDB syntax, the Query Store option will be on by default. For databases restored to SQL Server 2016 or later, the Query Store’s status from the original system will remain unchanged when the database is restored on the new instance.

Let’s go through the three ways to enable Query Store.

1. Manually in SQL Server Management Studio
2. Using T-SQL
3. Using PowerShell

1. Enabling Query Store using SQL Server Management Studio:

Since you’re likely already comfortable using SQL Server Management Studio for queries and database maintenance, SMSS does offer a convenient, familiar method for getting started with Query Store.

Steps to Enable Query Store using SSMS

1. Connect to a SQL Server instance running SQL Server 2016 or higher.
2. Click the ‘+’ sign next to the Databases folder to expand and see the list of databases.
3. Right-click on the database name and select “Properties.”
4. Left-click the “Query Store” option on the left-hand side of the GUI.
5. Change the Operation Mode(Requested) option from “Off” to “Read write.”
6. Click OK to apply the change and enable Query Store.

Further Details

Here is what you will see after step 3. The Query Store option mentioned in step 4 is at the bottom of the list of options, like the below.

Left-clicking that Query Store option will cause the below to show up on the right of the SSMS GUI.

What you see when you do that are the existing defaults on 2019 and above. If you are enabling Query Store on versions 2016 or 2017, you will want to adjust additional defaults. Prior to 2019, the default for “Query Store Capture Mode” was “All.” Change this option to “Auto” instead.

Furthermore, the default for Max_Storage_Size_MB was far too low in 2016 and 2017 and could be better in 2019 as well. This value represents the maximum amount of space that Query Store data will occupy in the database in which it was enabled. A good default value to start with is 2048 MB. It may be necessary to adjust that to 4096 MB at the high end in order to capture queries for the entire length of the “Stale Query Threshold (Days)” value.

The “Stale Query Threshold (Days)” option controls how many days of Query Store data will be kept. If the max storage size is set too low for a retention value of 30 days, then Query Store will start deleting collected data in the system tables to keep Query Store below the max storage size. This could result in having less data available than you intend for troubleshooting.

The rest of the defaults are acceptable and so could be left alone without concern.

2. Enabling Query Store using T-SQL

The T-SQL language is, of course, the language of SQL Server. It is often more flexible than the SSMS GUI. Notice in the screenshot up above that there is a “Script” button. If you click that instead of clicking “ok” in the UI, then SQL Server will script out the options in the GUI into a query window. This will allow you to see what the GUI does. Using T-SQL, it is easier to enable Query Store on multiple databases. You can use a construct like sp_msforeachdb to enable Query Store for multiple databases at once.

USE [master]
GO
ALTER DATABASE [MyDB] SET QUERY_STORE = ON
GO
ALTER DATABASE [MyDB] SET QUERY_STORE (OPERATION_MODE = READ_WRITE, MAX_STORAGE_SIZE_MB = 2048)
GO

3. Enabling Query Store using PowerShell

Many accidental DBAs, those folks who were “voluntold” to start managing SQL Server, are network engineers, sysadmins, or cloud admins. Automation is often music to their ears, and in the Windows universe, PowerShell is a go-to method for automating tasks. Consequently, using PowerShell to automate the enabling of Query Store may feel natural to accidental DBAs. For the below command, the DBATools module will be needed in your environment.

Below is how Query Store could be enabled on all user databases on an instance of SQL Server. If you only want to enable Query Store on a few select databases on an instance, then add the -Database parameter with a comma-separated list of databases.

Set-DbaDbQueryStoreOption -SqlInstance ServerA -State ReadWrite ​

-FlushInterval 900 -CollectionInterval 60 -MaxSize 4096 ​

-CaptureMode Auto -CleanupMode Auto -StaleQueryThreshold 30, -WaitStatsCaptureMode ON

Also, if your SQL Server environment has the Registered Server feature set up, then PowerShell can be used to read the servers registered there, loop over them, and enable Query Store on all user databases across your environment. This would be done using the Get-DbaRegServer command in the DBATools module.

Trace Flags for Query Store

If you aren’t familiar with Trace Flags, these are numbers that Microsoft uses to enable certain kinds of behavior in the database engine. They are occasionally meant to be short-term fixes, and later the functionality in a trace flag is built into how the SQL Server database engine works. This is the case for trace flags and Query Store. There are two trace flags to know about and enable. Notice that flag 7752 isn’t needed on SQL Server 2019 and above.

Trace Flag 7745—This prevents Query Store data from writing to disk prior to shutdown or failover process so it doesn’t delay a shutdown or failover.​

Trace Flag 7752 – Loads Query Store data to memory asynchronously from query execution. This default is built into the engine in SQL Server 2019.​

Want to Work With The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post How to Enable Query Store in SQL Server: A Step-by-Step Guide appeared first on The SERO Group.

What are the benefits of enabling Query Store? While there are many technical reasons, here are my top four broad advantages to consider.

1. Free SQL Server monitoring

Your business has already paid for Query Store in the SQL Server licensing. With SQL Server 2016 and later, it is accessible at the database level. This means that for smaller shops that may not have a large enterprise environment, you don’t have to spend large sums of money to get observability from 3rd party software. Query Store’s native capture mechanisms can provide significant insight into your SQL Server’s performance, all without costing you any more money!

2. Capture foundational SQL Server performance indicators

Query Store collects the data already present in your SQL Server, displaying it in easy-to-understand graphs and reports. With Query Store, values for CPU, memory, duration, and more can be viewed based on MAX/AVG/STD Deviation metrics per query. This provides valuable insights into core metrics that shape the performance of your applications. Furthermore, this data allows your company to see not only how specific queries behaved when there was a performance problem but also to trend those queries over time to see shifts in performance.

SQL Server wait statistics are also captured and displayed in Query Store. When a query needs a resource, like CPU, or data read from disk, then a wait type is assigned to the query. These various waits affect query performance in a multitude of ways, and Query Store surfaces those performance-impacting waits for you. For example, the Query Wait Statistics report may show large bar graphs for BUFFER IO and CPU. Queries appearing in both graphs may be suffering from large table scans because of missing indexes.

Additionally, Query Store captures the query plans associated with queries. Think of query plans as the blueprint for how the query will be executed. These plans contain data about the decisions SQL Server is making about your data and how to process it. Some decisions revealed in the query plan can pinpoint performance issues. For example, query plans that regularly contain table scan operators may indicate missing indexes that force SQL Server to scan millions of rows when it only needs to retrieve a few thousand rows.

3. Talk to your vendors with data in hand

COTS vendors need to see hard data when approached with a performance problem. Query Store can provide that data. Without it, you can report a problem, but the software vendor is unlikely to consider making changes.

If you engage a DBA as a Service company, having performance data in hand will go a long way toward building a good relationship with that vendor. They will see your preparedness and be drawn to that. Also, it will allow them to solve your problem faster, and isn’t that what you really want anyway?

4. Allow your applications to take advantage of new performance features

Newer versions of SQL Server have a collection of features known as Intelligent Query Processing (IQP). Features such as memory grant feedback, degree of parallelism feedback, and more are tied into IQP. These features depend on Query Store. Without Query Store running and without using the appropriate database compatibility level, your applications are missing out on performance-enhancing features that make queries execute faster, use fewer resources, or do both at the same time.

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post 4 Key Performance Benefits of Enabling Query Store appeared first on The SERO Group.