-- The query
SELECT 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1) order_date
from ss
group by 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1)

-- The query
SELECT 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1) order_date
from ss
group by 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1)

How to optimize this query?

The Problem of optimizing SQL query: Preventing Index Usage

The query is not SARGable (Searchable Argument-able). This means the database engine can’t use an index to find the data it needs directly. Instead, it must perform a full table scan, reading all billion rows from the disk, calculating the first day of the month for each one, and then grouping the results. This is incredibly slow and resource-intensive.

Optimization Strategies

Create a Composite Index – The Biggest Win

Regardless of how you rewrite the query, the most important step is to have the right index. Since you are grouping by SOLDTONBR and then by a derivative of ORDERDATE, the ideal index would cover both columns.

CREATE INDEX IX_ss_SoldToNbr_OrderDate ON ss (SOLDTONBR, ORDERDATE);

This index allows the database to quickly access the data pre-sorted by SOLDTONBR and ORDERDATE, making the grouping operation vastly more efficient. Optimizing SQL query.

Optimisation tips you can also find here: 15 Essential SQL Tips You Can’t Live Without

Column store indices for MS SQL Server you can find here

Use More Efficient Date Functions

Instead of combining YEAR and MONTH, use a single, more efficient function to truncate the date to the beginning of the month. The best function depends on your specific database system. These functions are often better optimized internally. Optimizing SQL query.

• SQL Server:SQL-- Best for modern SQL Server SELECT ss.SOLDTONBR, DATETRUNC(month, ORDERDATE) AS order_date FROM ss GROUP BY ss.SOLDTONBR, DATETRUNC(month, ORDERDATE);
• PostgreSQL / Redshift:SQLSELECT ss.SOLDTONBR, DATE_TRUNC('month', ORDERDATE) AS order_date FROM ss GROUP BY ss.SOLDTONBR, DATE_TRUNC('month', ORDERDATE);
• MySQL:SQLSELECT ss.SOLDTONBR, DATE_FORMAT(ORDERDATE, '%Y-%m-01') AS order_date FROM ss GROUP BY ss.SOLDTONBR, order_date; -- MySQL allows using alias in GROUP BY
• Oracle:SQLSELECT ss.SOLDTONBR, TRUNC(ORDERDATE, 'MM') AS order_date FROM ss GROUP BY ss.SOLDTONBR, TRUNC(ORDERDATE, 'MM');

While better, these still apply a function to the column, which can limit performance. The index from step 1 is still what makes the biggest difference.

3. The Gold Standard: A Date Dimension Table

For very large tables (like yours), the best practice is to use a date dimension table. This is a static lookup table containing one row for every day over a long period (e.g., 100 years). It has pre-calculated columns for everything you might need: first day of the month, week of the year, quarter, etc.

Here you can see how to create a Date Dimension Table using recursive CTE: How Works Recursive CTE in SQL Server?

SET DATEFIRST  1,
    DATEFORMAT ymd,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20220101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 5, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,

        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
      
)
SELECT
    dd.*,
    k.WeekOfYear_fdt
FROM
    DateDimension dd
LEFT JOIN
    (
        SELECT
            Year,
            WeekOfYear,
            MIN(Date) as WeekOfYear_fdt
        FROM DateDimension
        GROUP BY Year, WeekOfYear
    ) AS k ON k.Year = dd.Year AND k.WeekOfYear = dd.WeekOfYear
ORDER BY
    Date
OPTION (MAXRECURSION 0);

SET DATEFIRST  1,
    DATEFORMAT ymd,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20220101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 5, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,

        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
      
)
SELECT
    dd.*,
    k.WeekOfYear_fdt
FROM
    DateDimension dd
LEFT JOIN
    (
        SELECT
            Year,
            WeekOfYear,
            MIN(Date) as WeekOfYear_fdt
        FROM DateDimension
        GROUP BY Year, WeekOfYear
    ) AS k ON k.Year = dd.Year AND k.WeekOfYear = dd.WeekOfYear
ORDER BY
    Date
OPTION (MAXRECURSION 0);

Example DimDate Table:

How to Use It:

You join your main table to DimDate. The expensive GROUP BY calculation is replaced with a fast join and a group on a pre-calculated column.

SQL. Optimizing SQL query.

SELECT
    ss.SOLDTONBR,
    d.FirstOfMonth AS order_date
FROM
    ss
INNER JOIN
    DimDate d ON ss.ORDERDATE = d.FullDate -- Assumes ORDERDATE has no time component
GROUP BY
    ss.SOLDTONBR,
    d.FirstOfMonth;

Why this is the most efficient:

• No on-the-fly calculations: The grouping is done on a column that is already calculated (d.FirstOfMonth).
• Index-friendly: The join ss.ORDERDATE = d.FullDate can use indexes on both columns for maximum speed.
• Flexibility: It makes grouping by any other time period (week, quarter) trivial and just as fast.

Grouping by Week of the Year

To group by week, you can apply the same principles.

1. Function-based (Less Efficient):Use a function like DATETRUNC(‘week’, ORDERDATE) or DATEPART(week, ORDERDATE).SQL-- Example for PostgreSQL / SQL Server SELECT ss.SOLDTONBR, DATETRUNC(week, ORDERDATE) as order_week FROM ss GROUP BY ss.SOLDTONBR, DATETRUNC(week, ORDERDATE);
2. Date Dimension Table (Most Efficient):This is the ideal way. Just change the column in your GROUP BY clause.SQLSELECT ss.SOLDTONBR, d.WeekOfYear, -- Or a column representing the first day of the week d.Year FROM ss INNER JOIN DimDate d ON ss.ORDERDATE = d.FullDate GROUP BY ss.SOLDTONBR, d.WeekOfYear, d.Year;

Summary of Recommendations

For a table with billions of records, follow these steps for maximum performance (Optimizing SQL query.):

1. Immediately create a composite index: CREATE INDEX IX_ss_SoldToNbr_OrderDate ON ss (SOLDTONBR, ORDERDATE);
2. For the ultimate solution, create and populate a Date Dimension table. Join to this table to perform your aggregations. This is the most scalable and flexible method.
3. If you cannot create a dimension table, replace your DATEFROMPARTS(YEAR(...), MONTH(...)) logic with a more efficient, platform-specific function like DATETRUNC(). Optimizing SQL query.

The post Optimizing SQL query for billions of records table – a powerful tips appeared first on Customer Experience Management.

SQL is the backbone of data-driven decision-making, and mastering it can dramatically improve how you interact with databases. Whether you’re a seasoned developer or just starting out, writing efficient, readable, and scalable SQL queries is a skill that pays off daily. In this post, I’ve compiled ten essential tips that will help you write smarter SQL—tips that I’ve learned, refined, and shared over time. These aren’t just theoretical best practices; they’re practical techniques that can make your queries faster, your code cleaner, and your debugging easier.

You can try Microsoft SQL Server from here: https://www.microsoft.com/pl-pl/sql-server/sql-server-downloads. And developer edition is here

From avoiding SELECT * to choosing the right join types, each tip is designed to help you think critically about how your queries perform and how they scale. You’ll also learn how to use indexes effectively, filter data early, and make smart choices between EXISTS and IN. Each section includes a short summary and a link to a full post where you can dive deeper into the topic. Whether you’re optimizing a legacy system or building something new, these tips will help you get the most out of SQL—and avoid common pitfalls that slow down your work.

Select Only What You Need, SQL Tips no 1

Avoid SELECT * and specify only the columns you need. This reduces data transfer, memory usage, and improves query speed. For example, instead of pulling all employee data, just select employee_id, first_name, and last_name. Read more

Optimize Join Conditions for Performance

Avoid non-SARGable joins that prevent index usage. Instead of applying functions to columns in join conditions, restructure the logic to preserve index efficiency. This dramatically improves query speed. Read the full guide

Use the PIVOT Operator for Better Reporting

Transform row-based data into columnar format using PIVOT. This is ideal for cross-tab reports and trend analysis, especially when comparing metrics across time or categories. Explore the PIVOT tutorial

Master Recursive CTEs for Hierarchical Data

Recursive Common Table Expressions (CTEs) allow you to elegantly query hierarchical or tree-structured data. They’re powerful for tasks like organizational charts or category trees. Learn about recursive CTEs

Set the First Day of the Week with DATEFIRST

Use SET DATEFIRST to control how SQL Server interprets weekday numbers. This is crucial for accurate time-based reporting and week-based aggregations. See how to use DATEFIRST

Update Multiple Tables with Conditions

Learn how to structure multi-table updates using joins and conditional logic. This technique is essential for synchronizing data across related tables. Read the multi-table update example

Filter Early with WHERE Clauses

Apply filters as early as possible to reduce the number of rows processed in joins and aggregations. Optimize your filtering

Use UNION ALL Instead of UNION

UNION ALL is faster than UNION because it skips duplicate elimination. Use it when duplicates aren’t a concern. Performance tip explained

Avoid Functions on Indexed Columns

Using functions like LOWER() or DATEADD() on indexed columns disables index usage. Rewrite conditions to preserve index paths. Join optimization example

Explore SQL for Business Analytics

Advanced SQL techniques like statistical analysis, predictive modeling, and time series forecasting are essential for business analysts. Learn how to combine SQL with Python for deeper insights. Check out the full course

5 additional SQL tips

Use CTEs for Readability and Reuse

Common Table Expressions (CTEs) make complex queries easier to read and maintain. They allow you to define temporary result sets that can be referenced multiple times. SQL Tips

WITH recentorders AS (
  SELECT customerid, orderdate
  FROM orders
  WHERE orderdate > CURRENTDATE - INTERVAL '30 days'
)
SELECT customerid, COUNT(*) AS ordercount
FROM recentorders
GROUP BY customer_id;

Avoid Functions on Indexed Columns

Using functions on indexed columns disables index usage, slowing down queries. Instead, transform the value before comparison. SQL Tips

-- Avoid
SELECT FROM users WHERE LOWER(email) = 'test@example.com';
-- Better
SELECT FROM users WHERE email = 'test@example.com';

Use CASE for Conditional Logic

CASE lets you embed conditional logic directly in your queries, useful for categorizing or transforming data.

SELECT name,
  CASE
    WHEN score >= 90 THEN 'Excellent'
    WHEN score >= 75 THEN 'Good'
    ELSE 'Needs Improvement'
  END AS performance
FROM students;

Optimize Aggregations with GROUP BY

When aggregating data, ensure you’re grouping only necessary columns to avoid performance hits and incorrect results.

SELECT department, AVG(salary) AS avg_salary
FROM employees
GROUP BY department;

Use Parameterized Queries to Prevent SQL Injection

Always use parameterized queries in application code to protect against SQL injection.

-- Example in Python with psycopg2
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))

Remember – tips summary

Use Joins Efficiently

Choose the right join type—INNER JOIN for matched rows, and avoid CROSS JOIN unless necessary. Efficient joins reduce unnecessary data processing and improve clarity.

Filter Data Early

Apply filters as soon as possible in your query using WHERE clauses. This minimizes the number of rows processed in joins and aggregations, leading to faster execution. SQL Tips no 3

Use Indexes Wisely

Indexes speed up data retrieval, especially in WHERE, JOIN, and ORDER BY clauses. But don’t over-index—too many can slow down INSERT, UPDATE, and DELETE operations.

Avoid Subqueries in WHERE Clauses

Correlated subqueries can be slow. Replace them with joins when possible to improve performance and readability. SQL Tips no 4

Use UNION ALL Instead of UNION

UNION removes duplicates, which is costly. If duplicates aren’t a concern, use UNION ALL for faster results.

Limit Your Results

Use LIMIT or TOP to restrict the number of rows returned. This is especially useful for pagination or sampling large datasets.

Be Cautious with LIKE and Functions

Avoid leading wildcards in LIKE and functions in WHERE clauses—they prevent index usage. Instead, use indexed columns and consistent casing.

Use EXISTS Instead of IN

EXISTS is often faster than IN because it stops scanning once a match is found. Use it for subqueries checking row existence.

Use Appropriate Data Types

Choosing the right data type—like TINYINT over INT or CHAR over VARCHAR—can save space and improve performance.

SQL Server Management Studio

SSMS as a Comprehensive SQL Environment
SQL Server Management Studio (https://learn.microsoft.com/en-us/ssms/) is a powerful, integrated environment for managing SQL Server infrastructure. It provides tools for writing, executing, and debugging SQL queries, as well as managing databases, tables, views, and stored procedures. SSMS supports both on-premises and cloud-based SQL Server instances, making it versatile for hybrid environments. Its intuitive interface includes Object Explorer for navigating server components and Query Editor for crafting and testing SQL scripts. Whether you’re a database administrator or developer, SSMS offers a unified workspace that streamlines daily tasks and enhances productivity through built-in templates, syntax highlighting, and error diagnostics.

Security, Performance, and Monitoring Tools
SSMS includes robust features for security management, such as configuring roles, permissions, and auditing access. It also provides performance tuning tools like the Database Engine Tuning Advisor and graphical execution plans to help identify bottlenecks. With Activity Monitor, users can track real-time server performance, view active sessions, and analyze resource usage. These tools empower teams to maintain optimal database health and ensure compliance with organizational policies. SSMS also integrates with SQL Server Agent for scheduling jobs and alerts, making it a central hub for automation and proactive monitoring across enterprise environments.

Integration, Extensibility, and Collaboration
SSMS supports integration with source control systems like Git, enabling versioning and collaborative development. It allows exporting and importing data via wizards, scripting database objects, and generating reports for documentation. Users can extend SSMS functionality through add-ins and connect to Azure services for cloud-based analytics and storage. Its support for multiple query windows and tabbed editing enhances multitasking, while customizable keyboard shortcuts and themes improve user experience. SSMS continues to evolve with regular updates, ensuring compatibility with the latest SQL Server features and providing a stable platform for modern data operations.

The post 15 Essential SQL Tips You Can’t Live Without appeared first on Customer Experience Management.

A recursive CTE in SQL Server consists of:

1. Anchor member – the starting point.
2. Recursive member – repeatedly executed until a condition is met.

Recursive CTE in SQL Server
-- Prevent regional settings from interfering with date interpretation
SET DATEFIRST  7,         -- Set Sunday as the first day of the week
    DATEFORMAT mdy,       -- Set date format to Month-Day-Year
    LANGUAGE   US_ENGLISH;-- Use US English for language settings

-- Declare start and cutoff dates
DECLARE @StartDate  date = '20100101'; -- Start date: Jan 1, 2010
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));
-- Cutoff date: one day before Jan 1, 2013 (i.e., Dec 31, 2012)

-- Recursive CTE to generate a sequence of numbers from 0 to number of days between StartDate and CutoffDate
WITH seq(n) AS 
(
    SELECT 0               -- Anchor member: start with 0
    UNION ALL
    SELECT n + 1           -- Recursive member: increment n
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
    -- Stop when n reaches the number of days between StartDate and CutoffDate
)
SELECT n FROM seq          -- Select the generated sequence
ORDER BY n                 -- Order the result
OPTION (MAXRECURSION 0);   -- Allow unlimited recursion (default is 100)

Recursive CTE in SQL Server
-- Prevent regional settings from interfering with date interpretation
SET DATEFIRST  7,         -- Set Sunday as the first day of the week
    DATEFORMAT mdy,       -- Set date format to Month-Day-Year
    LANGUAGE   US_ENGLISH;-- Use US English for language settings

-- Declare start and cutoff dates
DECLARE @StartDate  date = '20100101'; -- Start date: Jan 1, 2010
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));
-- Cutoff date: one day before Jan 1, 2013 (i.e., Dec 31, 2012)

-- Recursive CTE to generate a sequence of numbers from 0 to number of days between StartDate and CutoffDate
WITH seq(n) AS 
(
    SELECT 0               -- Anchor member: start with 0
    UNION ALL
    SELECT n + 1           -- Recursive member: increment n
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
    -- Stop when n reaches the number of days between StartDate and CutoffDate
)
SELECT n FROM seq          -- Select the generated sequence
ORDER BY n                 -- Order the result
OPTION (MAXRECURSION 0);   -- Allow unlimited recursion (default is 100)

Recursive CTE in SQL Server.

Data table for BI

In business intelligence (BI) and data warehousing, a date dimension table is a foundational component that enables powerful time-based analysis. This SQL script demonstrates how to dynamically generate a comprehensive date dimension using a recursive Common Table Expression (CTE). Starting from a defined StartDate, it builds a sequence of dates up to a CutoffDate, enriching each date with attributes such as day name, month, quarter, week number, and weekend flag. This structure supports flexible filtering, grouping, and visualization in BI tools like Power BI, Tableau, or SQL Server Reporting Services (SSRS), and ensures consistent date logic across reports and dashboards.

Recursive CTE in SQL Server

1. Translate the generated numbers into actual dates.
2. Create a date dimension table suitable for use in BI tools like Power BI or Tableau.

Recursive CTE in SQL Server
-- Regional settings
SET DATEFIRST  7,
    DATEFORMAT mdy,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20100101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,
        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
)
SELECT * FROM DateDimension
ORDER BY Date
OPTION (MAXRECURSION 0);

Recursive CTE in SQL Server
-- Regional settings
SET DATEFIRST  7,
    DATEFORMAT mdy,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20100101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,
        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
)
SELECT * FROM DateDimension
ORDER BY Date
OPTION (MAXRECURSION 0);

Trad more here and How To Set The First Day of Week DATEFIRST

Next Steps

• Add fiscal year columns
• Add holiday flags (if you provide a list)
• Localize day/month names to Polish if needed

Fiscal Year Columns

Recursive CTE in SQL Server
-- Assuming fiscal year starts in April
SELECT
    Date,
    CASE 
        WHEN MONTH(Date) >= 4 THEN YEAR(Date)
        ELSE YEAR(Date) - 1
    END AS FiscalYear,
    CASE 
        WHEN MONTH(Date) BETWEEN 4 AND 6 THEN 1
        WHEN MONTH(Date) BETWEEN 7 AND 9 THEN 2
        WHEN MONTH(Date) BETWEEN 10 AND 12 THEN 3
        ELSE 4
    END AS FiscalQuarter
FROM DateDimension

Recursive CTE in SQL Server
-- Assuming fiscal year starts in April
SELECT
    Date,
    CASE 
        WHEN MONTH(Date) >= 4 THEN YEAR(Date)
        ELSE YEAR(Date) - 1
    END AS FiscalYear,
    CASE 
        WHEN MONTH(Date) BETWEEN 4 AND 6 THEN 1
        WHEN MONTH(Date) BETWEEN 7 AND 9 THEN 2
        WHEN MONTH(Date) BETWEEN 10 AND 12 THEN 3
        ELSE 4
    END AS FiscalQuarter
FROM DateDimension

Holiday Flags

It requires a holiday list. Assuming you have a table called HolidayList with a column HolidayDate.

Recursive CTE in SQL Server
SELECT
    d.*,
    CASE 
        WHEN h.HolidayDate IS NOT NULL THEN 1
        ELSE 0
    END AS IsHoliday
FROM DateDimension d
LEFT JOIN HolidayList h ON d.Date = h.HolidayDate

Recursive CTE in SQL Server
SELECT
    d.*,
    CASE 
        WHEN h.HolidayDate IS NOT NULL THEN 1
        ELSE 0
    END AS IsHoliday
FROM DateDimension d
LEFT JOIN HolidayList h ON d.Date = h.HolidayDate

Localize Day/Month Names to Polish

SELECT
    Date,
    CASE DATENAME(WEEKDAY, Date)
        WHEN 'Monday' THEN 'Poniedziałek'
        WHEN 'Tuesday' THEN 'Wtorek'
        WHEN 'Wednesday' THEN 'Środa'
        WHEN 'Thursday' THEN 'Czwartek'
        WHEN 'Friday' THEN 'Piątek'
        WHEN 'Saturday' THEN 'Sobota'
        WHEN 'Sunday' THEN 'Niedziela'
    END AS DayNamePL,
    CASE DATENAME(MONTH, Date)
        WHEN 'January' THEN 'Styczeń'
        WHEN 'February' THEN 'Luty'
        WHEN 'March' THEN 'Marzec'
        WHEN 'April' THEN 'Kwiecień'
        WHEN 'May' THEN 'Maj'
        WHEN 'June' THEN 'Czerwiec'
        WHEN 'July' THEN 'Lipiec'
        WHEN 'August' THEN 'Sierpień'
        WHEN 'September' THEN 'Wrzesień'
        WHEN 'October' THEN 'Październik'
        WHEN 'November' THEN 'Listopad'
        WHEN 'December' THEN 'Grudzień'
    END AS MonthNamePL
FROM DateDimension

SELECT
    Date,
    CASE DATENAME(WEEKDAY, Date)
        WHEN 'Monday' THEN 'Poniedziałek'
        WHEN 'Tuesday' THEN 'Wtorek'
        WHEN 'Wednesday' THEN 'Środa'
        WHEN 'Thursday' THEN 'Czwartek'
        WHEN 'Friday' THEN 'Piątek'
        WHEN 'Saturday' THEN 'Sobota'
        WHEN 'Sunday' THEN 'Niedziela'
    END AS DayNamePL,
    CASE DATENAME(MONTH, Date)
        WHEN 'January' THEN 'Styczeń'
        WHEN 'February' THEN 'Luty'
        WHEN 'March' THEN 'Marzec'
        WHEN 'April' THEN 'Kwiecień'
        WHEN 'May' THEN 'Maj'
        WHEN 'June' THEN 'Czerwiec'
        WHEN 'July' THEN 'Lipiec'
        WHEN 'August' THEN 'Sierpień'
        WHEN 'September' THEN 'Wrzesień'
        WHEN 'October' THEN 'Październik'
        WHEN 'November' THEN 'Listopad'
        WHEN 'December' THEN 'Grudzień'
    END AS MonthNamePL
FROM DateDimension

This extended SQL script builds a robust date dimension table tailored for business intelligence applications. Starting with a recursive CTE to generate a sequence of dates, it enriches each entry with key calendar attributes such as day names, month names, quarters, and weekend flags. Additional enhancements include fiscal year logic, holiday indicators (based on a provided list), and localization of day and month names to Polish. These features ensure the table supports flexible, localized, and context-aware time-based analysis in tools like Power BI, Tableau, or SQL Server Reporting Services.

The post How Works Recursive CTE in SQL Server? appeared first on Customer Experience Management.

Reduce records in reference. Instead of query 1 which is quite heavy you can use query 2 – much lighter.

-- query 1
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

-- query 1
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

-- query 2
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

-- query 2
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

10 Tips How To Make SQL Lighter

1. Select Only What You Need

Instead of using SELECT *, specify the exact columns you need. This reduces the amount of data transferred from the database server to the application, lowering network traffic and memory usage.

SQL

-- Bad
SELECT * FROM employees;

-- Good
SELECT employee_id, first_name, last_name FROM employees;

2. Use Joins Efficiently

Use the most appropriate join type for your needs. For example, use INNER JOIN when you only need rows with matching values in both tables. Avoid cross joins if possible, as they produce the Cartesian product of the two tables, which can be massive.

3. Filter Data Early. 10 Tips How To Make SQL Lighter.

Use the WHERE clause to filter data as early as possible. This reduces the number of rows processed by subsequent operations like joins or aggregations.

SQL

-- Bad
SELECT first_name, last_name, salary
FROM employees
WHERE department_id IN (
    SELECT department_id FROM departments WHERE location = 'New York'
);

-- Good (filtering early)
SELECT e.first_name, e.last_name, e.salary
FROM employees AS e
JOIN departments AS d ON e.department_id = d.department_id
WHERE d.location = 'New York';

4. Use Indexes

Indexes are crucial for speeding up data retrieval. Ensure that columns used in WHERE clauses, JOIN conditions, and ORDER BY clauses are indexed. However, be mindful that too many indexes can slow down INSERT, UPDATE, and DELETE operations.

5. Avoid Subqueries in the WHERE Clause

Correlated subqueries can be very slow because they execute once for each row in the outer query. It’s often more efficient to rewrite them as JOINs.

6. Use UNION ALL Instead of UNION

UNION removes duplicate rows, which is an expensive operation. If you know there are no duplicates or you don’t need to remove them, use UNION ALL for better performance.

7. Limit Your Results

When fetching a large dataset, use LIMIT (or TOP in SQL Server) to restrict the number of rows returned. This is especially useful for pagination or when you only need a sample of the data. 10 Tips How To Make SQL Lighter.

8. Be Cautious with LIKE and Functions

Using the LIKE operator with a leading wildcard (%) or applying functions to columns in a WHERE clause can prevent the database from using an index. Instead of '%value', use 'value%'. Avoid WHERE UPPER(column) = 'VALUE', and try to store data in a consistent case.

9. Use EXISTS Instead of IN

For subqueries that check for the existence of rows, EXISTS is often faster than IN because EXISTS stops scanning as soon as it finds a match, whereas IN might scan the entire subquery result set.

10. Use Appropriate Data Types

Choosing the right data type for your columns can save storage space and improve performance. For example, using TINYINT instead of INT for a small numerical range, or CHAR instead of VARCHAR if the length is fixed.

Some tips consulted with https://gemini.google.com

More sql tips here

The post 10 Tips How To Make SQL Lighter appeared first on Customer Experience Management.

Python’s powerful date and time functions using the datetime and pandas libraries gives you a robust date table ready for Power BI and other business intelligence and analytical tools. Python for analysts most important datetime functions.

Mastering Date and Time Functions in Python for Power BI Date Tables

When working with Power BI, a well-structured Date Table is essential for time intelligence calculations like YTD, QTD, MTD, and custom period comparisons. While Power BI has built-in date table features, using Python to generate a custom date table gives you full control over the structure, granularity, and logic.

In this post, we’ll explore Python’s powerful date and time functions using the datetime and pandas libraries, and show how to create a robust date table ready for Power BI.

Python for analyst – date and time functions – basics

Python provides the datetime module to work with dates and times. Here’s a quick overview:

from datetime import datetime, timedelta, date

# Current date and time
now = datetime.now()
print("Now:", now)

# Just the date
today = date.today()
print("Today:", today)

# Add 7 days
next_week = today + timedelta(days=7)
print("Next week:", next_week)

# Subtract 30 days
last_month = today - timedelta(days=30)
print("30 days ago:", last_month)

These functions are the foundation for generating date ranges and calculating custom columns like fiscal periods or holidays. Python for analysts most important datetime functions.

Creating a Date Range with Pandas

To build a date table, we need a continuous range of dates. pandas.date_range() is perfect for this:

import pandas as pd

# Generate a date range from 2020 to 2030
date_range = pd.date_range(start='2020-01-01', end='2030-12-31', freq='D')
df = pd.DataFrame({'Date': date_range})

This gives us a DataFrame with one row per day — the backbone of our date table.

Enriching the Date Table

Now let’s add useful columns for Power BI:

df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['MonthName'] = df['Date'].dt.strftime('%B')
df['Quarter'] = df['Date'].dt.quarter
df['Day'] = df['Date'].dt.day
df['Weekday'] = df['Date'].dt.weekday + 1  # Monday = 1
df['WeekdayName'] = df['Date'].dt.strftime('%A')
df['IsWeekend'] = df['Weekday'].isin([6, 7])
df['Week'] = df['Date'].dt.isocalendar().week
df['DayOfYear'] = df['Date'].dt.dayofyear

These columns allow for slicing and dicing your data in Power BI by year, month, weekday, and more.

Fiscal Calendar Support

Many businesses use fiscal calendars that don’t align with the calendar year. Here’s how to add a fiscal year starting in July:

df['FiscalYear'] = df['Date'].apply(lambda x: x.year if x.month < 7 else x.year + 1)
df['FiscalMonth'] = df['Date'].apply(lambda x: x.month - 6 if x.month >= 7 else x.month + 6)
df['FiscalQuarter'] = ((df['FiscalMonth'] - 1) // 3) + 1

This logic adjusts the fiscal year, month, and quarter based on a July start. Python for analysts most important datetime functions

5. Flags for Time Intelligence

Power BI benefits from flags that simplify DAX calculations:

today = pd.to_datetime('today').normalize()

df['IsToday'] = df['Date'] == today
df['IsCurrentMonth'] = (df['Date'].dt.month == today.month) & (df['Date'].dt.year == today.year)
df['IsCurrentYear'] = df['Date'].dt.year == today.year

You can also add flags for holidays, fiscal periods, or custom business logic. Python for analysts most important datetime functions

Exporting to CSV for Power BI

Once your date table is ready, export it:

df.to_csv('DateTable.csv', index=False)

You can now import this CSV into Power BI as a static date table.

7. Sample Output

Here’s a preview of what your date table might look like:

Pandas to_datetime() function

Python for analysts most important datetime functions – pandas

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  object        
 3   Dt_last_purchase        13483 non-null  object        

df_cust['Dt_first_purchase'] = pd.to_datetime(df_cust['Dt_first_purchase'],format="yyyy-mm-dd")
df_cust['Dt_last_purchase'] = pd.to_datetime(df_cust['Dt_last_purchase'],format="yyyy-mm-dd")

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  datetime64[ns]
 3   Dt_last_purchase        13483 non-null  datetime64[ns]

Advanced Tips

• Holidays: Use external APIs or CSVs to mark public holidays.
• Week Start: Adjust Weekday to match your locale (e.g., Monday vs. Sunday).
• Time Zones: Use pytz or zoneinfo for timezone-aware datetime handling.
• Dynamic Updates: Automate the script to regenerate the table monthly or yearly.

Certainly! Here’s a concise 400-word post on SQL Date and Time Functions, with examples, tailored for building a Date Table in Power BI:

SQL Date and Time Functions for Power BI Date Tables

When building reports in Power BI, a comprehensive Date Table is essential for enabling time-based calculations like YTD, MTD, and custom period comparisons. While Power BI can auto-generate a date table, using SQL to create one gives you full control over its structure and logic.

Let’s explore key SQL Server date and time functions and how to use them to build a robust date table.

1. Generating a Date Range

To create a date table, you need a continuous range of dates. In SQL Server, you can use a loop or a recursive CTE:

DECLARE @StartDate DATE = '2020-01-01';
DECLARE @EndDate DATE = '2030-12-31';

WITH DateCTE AS (
    SELECT @StartDate AS DateValue
    UNION ALL
    SELECT DATEADD(DAY, 1, DateValue)
    FROM DateCTE
    WHERE DateValue < @EndDate
)
SELECT * INTO DateTable FROM DateCTE
OPTION (MAXRECURSION 32767);

select * from DateTable

Here the example

This creates a table with one row per day between 2020 and 2030.

2. Adding Date Attributes

Once you have the base dates, enrich them with useful columns:

ALTER TABLE DateTable ADD 
    Year INT,
    Month INT,
    MonthName VARCHAR(20),
    Quarter INT,
    Weekday INT,
    WeekdayName VARCHAR(20);

UPDATE DateTable
SET 
    Year = YEAR(DateValue),
    Month = MONTH(DateValue),
    MonthName = DATENAME(MONTH, DateValue),
    Quarter = DATEPART(QUARTER, DateValue),
    Weekday = DATEPART(WEEKDAY, DateValue),
    WeekdayName = DATENAME(WEEKDAY, DateValue);

These columns allow for flexible filtering and grouping in Power BI.

3. Fiscal Calendar and Flags

You can also add fiscal logic and flags:

ALTER TABLE DateTable ADD FiscalYear INT;

UPDATE DateTable
SET FiscalYear = CASE 
    WHEN MONTH(DateValue) >= 7 THEN YEAR(DateValue) + 1
    ELSE YEAR(DateValue)
END;

Add flags like IsWeekend, IsToday, or IsCurrentMonth to simplify DAX expressions.

Conclusion

Python offers a flexible and powerful way to create a custom date table for Power BI. With just a few lines of code, you can generate a rich dataset that supports advanced time intelligence and reporting needs.

Whether you’re working with fiscal calendars, custom flags, or multilingual support, Python gives you the tools to tailor your date table exactly to your business requirements.

SQL’s date and time functions like DATEADD, DATEPART, DATENAME, and YEAR are powerful tools for building a custom date table. Once created, export it to Power BI or use it as a view for dynamic reporting.

Would you like a ready-to-run SQL script for a complete date table?

Find more resources in our course of Advanced programming for business analysts

The post Python for analysts most important datetime functions appeared first on Customer Experience Management.

Post is based on: Calculate a Moving Average with T-SQL Windowing Functions

-- test table
-- mssqltips.com
DROP TABLE IF EXISTS dbo.BigWeightTracker;
GO

CREATE TABLE dbo.BigWeightTracker
(
    Id INT IDENTITY(1, 1) NOT NULL,
    UserId INT NOT NULL,
    Pounds Decimal(10, 2) NOT NULL,
    DateRecorded DATE NOT NULL,
    CONSTRAINT PK_BigWeightTracker_Id
        PRIMARY KEY CLUSTERED (Id)
);
GO

DECLARE @StartDate DATE = '2022-01-01';
DECLARE @EndDate DATE = '2023-12-31';

WITH 
 Users AS (SELECT TOP 60
        ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS UserId
    -- select top 100 *
    FROM sys.all_columns AS s1
   ),
 Dates AS (
    SELECT @StartDate AS Date
    UNION ALL
    SELECT DATEADD(DAY, 1, Date) AS Date FROM Dates  WHERE Date < @EndDate
   )
--select * from Dates, ale insert działa bez tego ograniczenia

INSERT INTO dbo.BigWeightTracker
(
    UserId,
    DateRecorded,
    Pounds
)
SELECT u.UserId,
       d.Date,
       (ABS(CHECKSUM(NEWID()) % (220 - 170 + 1)) + 170) AS Weight
FROM Users u
    CROSS JOIN Dates d
OPTION (MAXRECURSION 0);
GO

-- koniec tworzenia tabeli -----------

Indexing

-- indexing
DROP INDEX IF EXISTS IX_BigWeightTracker_Average ON dbo.BigWeightTracker;

CREATE NONCLUSTERED INDEX IX_BigWeightTracker_Average
ON dbo.BigWeightTracker
(
    UserId,
    DateRecorded ASC
)
INCLUDE (Pounds);
GO

moving average – old approach

-- moving avg - old aproach
-- mssqltips.com
SET STATISTICS TIME, IO ON;
SELECT t1.UserId,
       t1.DateRecorded,
       t1.Pounds,
       CASE
           WHEN COUNT(t2.Pounds) < 7 THEN
               NULL
           ELSE
               AVG(t2.Pounds)
       END AS [7DayMovingAverage]
FROM dbo.BigWeightTracker t1
    INNER JOIN dbo.BigWeightTracker t2
        ON t2.DateRecorded
           BETWEEN DATEADD(DAY, -6, t1.DateRecorded) AND t1.DateRecorded
           AND t1.UserId = t2.UserId
GROUP BY t1.UserId,
         t1.DateRecorded,
         t1.Pounds
ORDER BY t1.UserId,
         t1.DateRecorded;
SET STATISTICS TIME, IO OFF;
GO

Statistics

(438000 rows affected)
Tabela „BigWeightTracker”. 
Liczba skanowań: 10, odczyty logiczne: 3310, odczyty z wyprzedzeniem: 1542.

Tabela „Worktable”. Liczba skanowań: 600, odczyty logiczne: 2634596, odczyty z wyprzedzeniem: 10957.
SQL Server — czasy wykonywania:
  Czas procesora CPU = 94376 ms; upłynęło czasu = 27683 ms.

Another way to write this query is to use a correlated subquery (podzapytanie). Based on the performance metrics, this approach still results in a high number of logical reads and scans with about the same execution time. Moving average in SQL.

-- qwerenda z podzapytaniem
-- mssqltips.com
SET STATISTICS TIME, IO ON;
SELECT t1.UserId,
       t1.DateRecorded,
       t1.Pounds,
       (
           SELECT CASE
                      WHEN COUNT(t2.Pounds) < 7 THEN
                          NULL
                      ELSE
                          AVG(t2.Pounds)
                  END
           FROM BigWeightTracker t2
           WHERE t2.DateRecorded
                 BETWEEN DATEADD(DAY, -6, t1.DateRecorded) AND t1.DateRecorded
                 AND t1.UserId = t2.UserId
       ) AS [7DayMovingAverage]
FROM dbo.BigWeightTracker t1
ORDER BY t1.UserId,
         t1.DateRecorded;
SET STATISTICS TIME, IO OFF;

Statystyka

(438000 rows affected)
Tabela „BigWeightTracker”. Liczba skanowań: 6, odczyty logiczne: 3232, odczyty fizyczne: 3, odczyty z wyprzedzeniem: 1573.
Tabela „Worktable”. Liczba skanowań: 1314000, odczyty logiczne: 6884158.
 SQL Server — czasy wykonywania:
  Czas procesora CPU = 8593 ms; upłynęło czasu = 5162 ms.

Window functions

Microsoft introduced window functions in SQL Server 2005, but added new and improved options in SQL Server 2012. A few functions to use with windowing include:

• ROW_NUMBER: Adds a sequential number to each row.
• RANK: Ranks rows and leaves a gap when there are ties (rows values equal).
• DENSE_RANK: Like RANK but doesn’t leave gaps when there are ties.

The OVER clause allows you to define your window frame. Additionally, you can tell the frame how many prior rows to include with the ROWS PRECEDING argument. Before we build the 7-day average, let’s look at a simple example. Moving average in SQL.

-- simple test
-- mssqltips.com
DECLARE @Simple_Test AS TABLE (Id INT);
INSERT INTO @Simple_Test
(Id) VALUES (1),(2),(3),(4),(5);

SELECT Id,
       SUM(Id) OVER (ORDER BY Id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS Value
FROM @Simple_Test;
GO

To achieve moving average, use the argument ROWS BETWEEN 1 PRECEDING AND CURRENT ROW.

-- mssqltips.com
SET STATISTICS TIME, IO ON;
SELECT t1.UserId,
       t1.DateRecorded,
       t1.Pounds,
       AVG(t1.Pounds) 
         OVER (
           PARTITION BY t1.UserId 
              ORDER BY t1.DateRecorded 
                 ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS [7DayMovingAverage]
FROM dbo.BigWeightTracker t1
ORDER BY t1.UserId,
         t1.DateRecorded;
SET STATISTICS TIME, IO OFF;

Statistics. Moving average in SQL

(438000 rows affected)
Tabela „BigWeightTracker”. Liczba skanowań: 5, odczyty logiczne: 1655,

 SQL Server — czasy wykonywania:
  Czas procesora CPU = 3079 ms; upłynęło czasu = 2382 ms.

Key Points of Moving average in SQL

• Windowing functions are an excellent solution for use cases like moving average, where you might otherwise perform self-joins or correlated subqueries to pull back the results.
• Additional arguments use in windowing function ROWS BETWEEN n PRECEDING AND CURRENT ROW.

Anomaly Detection for User Sessions and Logins using Moving average in SQL

-- ===================================================================================
-- Anomaly Detection for User Sessions and Logins
--
-- Purpose: This query identifies days with unusual session or login counts.
-- Method:  It flags a day as an anomaly if its count is more than 2 standard
--          deviations away from the 5-day moving average.
-- ===================================================================================

WITH RunningStats AS (
    -- This CTE calculates a 5-day moving average and standard deviation for session
    -- and logged-in user counts. The window includes the current day and the four previous days.
    SELECT
        [Probe_date],
        [User_internet],
        [User_logged],
        [Sesion_count],
        AVG(CAST(Sesion_count AS FLOAT)) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_avg_session,
        STDEV(Sesion_count) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_std_session,
        AVG(CAST(User_logged AS FLOAT)) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_avg_logged,
        STDEV(User_logged) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_std_logged
    FROM
        [DBSession_stat]
    -- Restrict the analysis to the last 30 days for relevance.
    WHERE
        [Probe_date] >= DATEADD(day, -30, GETUTCDATE())
)
-- Final step: Select all calculated stats but only for the days flagged as anomalous.
SELECT
    *
FROM
    RunningStats
-- The WHERE clause applies the "2-sigma" rule to filter for anomalies. A day is
-- returned if either the session count or the logged-in user count is anomalous.
WHERE
    -- Condition for session anomaly
    ABS(Sesion_count - moving_avg_session) > 2 * moving_std_session
    OR
    -- Condition for logged-in user anomaly
    ABS(User_logged - moving_avg_logged) > 2 * moving_std_logged;

Here another approach comparing also weekly pattern.

import pandas as pd

# --- 1. Load and Prepare the Data ---

# Load the data from the uploaded CSV file
df = pd.read_csv('Sesion_stat_last_30_days_an_sessions.xlsx - Arkusz4.csv')

# Convert 'Probe_date' to datetime objects for time-based analysis
df['Probe_date'] = pd.to_datetime(df['Probe_date'])

# Set the 'Probe_date' as the index of the DataFrame
df = df.set_index('Probe_date').sort_index()


# --- 2. Feature Engineering: Create Baselines for Comparison ---

# Define the number of data points in a week (7 days * 24 hours * 6 10-minute intervals)
# This is used to look back at the same time last week.
PREVIOUS_WEEK_SHIFT = 7 * 24 * 6

# Calculate the 1-hour running average (6 data points * 10 minutes = 60 minutes)
df['running_avg'] = df['Sesion_count'].rolling(window=6, min_periods=1).mean()

# Get the session count from the exact same time one week prior
df['prev_week_sessions'] = df['Sesion_count'].shift(PREVIOUS_WEEK_SHIFT)


# --- 3. Anomaly Detection Logic ---

# Define the thresholds for what constitutes an anomaly
# A 50% increase over both the running average and the previous week's value
RELATIVE_THRESHOLD = 1.5
# The session count must also be at least this high to be considered
ABSOLUTE_THRESHOLD = 50

# Apply the anomaly detection conditions
df['is_anomaly'] = (
    (df['Sesion_count'] > (df['running_avg'] * RELATIVE_THRESHOLD)) &
    (df['Sesion_count'] > (df['prev_week_sessions'] * RELATIVE_THRESHOLD)) &
    (df['Sesion_count'] > ABSOLUTE_THRESHOLD)
)


# --- 4. Review the Results ---

# Select only the rows where an anomaly was detected
anomalies = df[df['is_anomaly']].copy()

# Add columns to show the deviation that triggered the flag
anomalies['pct_over_avg'] = (anomalies['Sesion_count'] / anomalies['running_avg'] - 1) * 100
anomalies['pct_over_prev_week'] = (anomalies['Sesion_count'] / anomalies['prev_week_sessions'] - 1) * 100


print("Anomalies Detected:")
# Display the identified anomalies with context
print(anomalies[[
    'Sesion_count',
    'running_avg',
    'prev_week_sessions',
    'pct_over_avg',
    'pct_over_prev_week'
]].round(1))

Program detected following anomalies

Moving average in SQL

Read more Python for analysts most important datetime functions

The post Moving average in SQL appeared first on Customer Experience Management.

Course Structure:

Week 1-2: Introduction and Setup

• Lecture: Overview of SQL and Python in business analytics.
• Lab: Setting up the environment (SQL databases, Python IDEs).
• Project 1: Data Import and Cleaning
  • SQL: Importing data from various sources, cleaning and preprocessing.
  • Python: Using pandas for data cleaning and manipulation.

Week 3-4: Data Exploration and Visualization

• Lecture: Techniques for data exploration and visualization.
• Lab: SQL queries for data exploration, Python libraries for visualization (matplotlib, seaborn).
• Project 2: Exploratory Data Analysis (EDA)
  • SQL: Writing complex queries to explore data.
  • Python: Visualizing data trends and patterns.

Week 5-6: Statistical Analysis

• Lecture: Statistical methods for business analytics.
• Lab: SQL functions for statistical analysis, Python libraries (numpy, scipy).
• Project 3: Statistical Analysis of Sales Data
  • SQL: Calculating statistical measures (mean, median, standard deviation).
  • Python: Performing hypothesis testing and regression analysis.

Week 7-8: Predictive Modeling

• Lecture: Introduction to predictive modeling techniques.
• Lab: SQL for data preparation, Python for model building (scikit-learn).
• Project 4: Predictive Sales Forecasting
  • SQL: Preparing data for modeling.
  • Python: Building and evaluating predictive models.

Week 9-10: Time Series Analysis

• Lecture: Time series analysis and forecasting.
• Lab: SQL for time series data manipulation, Python libraries (statsmodels).
• Project 5: Time Series Forecasting
  • SQL: Extracting and transforming time series data.
  • Python: Building time series models and forecasting.

Week 11-12: Machine Learning Integration

• Lecture: Integrating machine learning with SQL databases.
• Lab: SQL for data storage and retrieval, Python for machine learning (TensorFlow, Keras).
• Project 6: Customer Segmentation
  • SQL: Storing and retrieving data for machine learning.
  • Python: Building and deploying machine learning models.

Week 13-14: Advanced Topics and Final Project

• Lecture: Advanced topics in SQL and Python (optimization, big data).
• Lab: SQL performance tuning, Python for big data (PySpark).
• Project 7: Comprehensive Business Analytics Project
  • SQL: Optimizing queries for large datasets.
  • Python: Analyzing and visualizing large datasets.

Assessment:

• Projects: Each mini project will be assessed based on accuracy, efficiency, and creativity.
• Final Project: A comprehensive project integrating all learned skills.

Resources:

• Books: “SQL for Data Analytics” by Upom Malik, Matt Goldwasser, and Benjamin Johnston; “Python for Data Analysis” by Wes McKinney.
• Online Tutorials: SQLZoo, DataCamp, Coursera.

Advanced Programming in SQL and Python

Advanced Programming in Data Analysis

https://datascience.umcs.pl

The post Advanced Programming in SQL and Python – the course for business analysts appeared first on Customer Experience Management.

1. First Normal Form (1NF): Removes repeating groups and ensures atomic values in each column.
2. Second Normal Form (2NF): Removes partial dependencies by ensuring all non-key columns depend on the full primary key.
3. Third Normal Form (3NF): Eliminates transitive dependencies, ensuring non-key columns depend only on primary keys.
4. Boyce-Codd Normal Form (BCNF): Strengthens 3NF by removing anomalies that exist due to overlapping candidate keys. If one column depends on some other column, then that “other column” must be enough to identify each row uniquely.
5. Fourth Normal Form (4NF): Ensures that a table does not mix multiple independent one-to-many relationships for the same entity.

The post Database Normal Forms Every Developer Should Know appeared first on Customer Experience Management.

When to Use SET DATEFIRST

Regional Settings: Use SET DATEFIRST to match the first day of the week with regional or cultural preferences. For example, in many European countries, Monday is considered the first day of the week.

• Consistent Reporting: Ensure consistency in date-related calculations and reports, especially when working with international data or users from different regions.
• Custom Calculations: When you need specific calculations that depend on the first day of the week, such as weekly summaries or reports.

How To Set The First Day of Week DATEFIRST

Here is the code which generate

-- Declare and initialize variables
DECLARE @i INT = 0;                  -- Loop counter
DECLARE @dt DATE;                   -- Current date in loop
DECLARE @dt_blost DATE;            -- Adjusted date based on weekday
DECLARE @tt TABLE (                -- Table variable to store results
    id INT IDENTITY,              -- Auto-incrementing ID
    dt DATE,                      -- Original date
    wd INT,                       -- Weekday number
    dt_blost DATE                 -- Adjusted date
);
DECLARE @wd INT;                   -- Weekday number

-- Set the first day of the week to Monday (1 = Monday, 7 = Sunday)
SET DATEFIRST 1;

-- Initialize @dt with the current date
SET @dt = GETDATE();

-- Loop 20 times to generate dates and adjusted dates
WHILE @i < 20
BEGIN
    -- Add @i days to the current date
    SET @dt = DATEADD(DAY, @i, @dt);

    -- Get the weekday number of the new date
    SET @wd = DATEPART(WEEKDAY, @dt);

    -- Calculate adjusted date (@dt_blost) based on weekday:
    -- If Monday (1), subtract 3 days
    -- If Tuesday (2), subtract 1 day
    -- Otherwise, add 2 days
    SET @dt_blost = DATEADD(DAY,
        IIF(@wd = 1, -3,
            IIF(@wd = 2, -1, 2)), @dt);

    -- Insert the date, weekday, and adjusted date into the table
    INSERT @tt
    SELECT @dt AS dt, @wd AS wd, @dt_blost;

    -- Increment loop counter
    SET @i += 1;
END

-- Display the results
SELECT * FROM @tt;

-- Additional check: show current UTC date, DATEFIRST setting,
-- weekday number and name for the current UTC date
SET DATEFIRST 1;
SELECT 
    GETUTCDATE() AS [getutcdate],
    @@DATEFIRST AS [DATEFIRST],
    DATEPART(WEEKDAY, GETUTCDATE()) AS [datepart_WEEKDAY],
    DATENAME(WEEKDAY, GETUTCDATE()) AS [datename_WEEKDAY];

Here’s a code in action

Explanation

1. Variable Declarations: The code declares and initializes several variables to store dates, weekday numbers, and a temporary table to hold the results.
2. SET DATEFIRST: This sets the first day of the week to Monday, which affects the output of date-related functions.
3. Loop: The loop runs 20 times, incrementing the date by the counter value each time.
4. Date Adjustments: Based on the weekday, the code adjusts the date:
   • If the date is Sunday (weekday 1), it subtracts 3 days.
   • If the date is Monday (weekday 2), it subtracts 1 day.
   • For other weekdays, it adds 2 days.
5. Insert into Temporary Table: The current date, weekday number, and adjusted date are inserted into the temporary table.
6. Select from Temporary Table: The code selects all records from the temporary table to display the results.
7. Final Select: The code selects the current UTC date, the DATEFIRST setting, the weekday number, and the weekday name to show the current settings and date information.

This script is useful for generating a series of dates, adjusting them based on specific rules, and storing the results for further analysis. If you have any more questions or need further clarification, feel free to ask!

You can read about date and time most important functions in post Python for analysts most important datetime functions

SQL Date and Time Functions for Power BI Date Tables

When building reports in Power BI, a comprehensive Date Table is essential for enabling time-based calculations like YTD, MTD, and custom period comparisons. While Power BI can auto-generate a date table, using SQL to create one gives you full control over its structure and logic.

Let’s explore key SQL Server date and time functions and how to use them to build a robust date table.

Generating a Date Range

To create a date table, you need a continuous range of dates. In SQL Server, you can use a loop or a recursive CTE:

DECLARE @StartDate DATE = '2020-01-01';
DECLARE @EndDate DATE = '2030-12-31';

WITH DateCTE AS (
    SELECT @StartDate AS DateValue
    UNION ALL
    SELECT DATEADD(DAY, 1, DateValue)
    FROM DateCTE
    WHERE DateValue < @EndDate
)
SELECT * INTO DateTable FROM DateCTE
OPTION (MAXRECURSION 32767);

This creates a table with one row per day between 2020 and 2030.

Alternate code you can find here How Works Recursive CTE in SQL Server?

Adding Date Attributes

Once you have the base dates, enrich them with useful columns:

ALTER TABLE DateTable ADD 
    Year INT,
    Month INT,
    MonthName VARCHAR(20),
    Quarter INT,
    Weekday INT,
    WeekdayName VARCHAR(20);

UPDATE DateTable
SET 
    Year = YEAR(DateValue),
    Month = MONTH(DateValue),
    MonthName = DATENAME(MONTH, DateValue),
    Quarter = DATEPART(QUARTER, DateValue),
    Weekday = DATEPART(WEEKDAY, DateValue),
    WeekdayName = DATENAME(WEEKDAY, DateValue);

These columns allow for flexible filtering and grouping in Power BI.

Fiscal Calendar and Flags

You can also add fiscal logic and flags:

ALTER TABLE DateTable ADD FiscalYear INT;

UPDATE DateTable
SET FiscalYear = CASE 
    WHEN MONTH(DateValue) >= 7 THEN YEAR(DateValue) + 1
    ELSE YEAR(DateValue)
END;

Add flags like IsWeekend, IsToday, or IsCurrentMonth to simplify DAX expressions.

Conclusion

SQL’s date and time functions like DATEADD, DATEPART, DATENAME, and YEAR are powerful tools for building a custom date table. Once created, export it to Power BI or use it as a view for dynamic reporting.

Would you like a ready-to-run SQL script for a complete date table?

Post written with the help of https://copilot.microsoft.com/

The post How To Set The First Day of Week DATEFIRST appeared first on Customer Experience Management.

Here’s a simple example of using the `PIVOT` command in SQL Server. Suppose you have a table named `Sales` with the following structure:

| Product | Quarter | SalesAmount |
|---------|---------|-------------|
| A | Q1 | 100 |
| A | Q2 | 150 |
| A | Q3 | 200 |
| A | Q4 | 250 |
| B | Q1 | 120 |
| B | Q2 | 180 |
| B | Q3 | 240 |
| B | Q4 | 300 |

You want to transform this data to show total sales for each product per quarter as columns. Here’s how you can achieve that using the `PIVOT` command:

SELECT Product, [Q1], [Q2], [Q3], [Q4]
FROM ( SELECT Product, Quarter, SalesAmount FROM Sales ) AS SourceTable
PIVOT ( SUM(SalesAmount) FOR Quarter IN ([Q1], [Q2], [Q3], [Q4]) ) AS PivotTable;

In this query (The PIVOT operator in SQL):

• 1. The subquery SourceTable selects the relevant columns from the Sales table.
• 2. The PIVOT operation specifies that we want to aggregate the SalesAmount using the SUM function, and change the Quarter values (Q1, Q2, Q3, Q4) into columns.
• 3. The final result shows total sales for each product across the quarters, with the quarter names as column headers.

The result will be:

| Product | Q1 | Q2 | Q3 | Q4 |
|---------|-----|-----|-----|-----|
| A | 100 | 150 | 200 | 250 |
| B | 120 | 180 | 240 | 300 |

Employing PIVOT involves specifying an aggregation function and a column source from which values will be extracted. A typical use case requires defining the columns to be pivoted and the aggregation logic, such as SUM or AVG, that will be applied to them. This operation facilitates the summarization of large datasets, aiding in data interpretation by highlighting trends, anomalies, and insights that would otherwise remain obscured in traditional row-oriented datasets. It effectively reduces the cognitive load required to analyze and interpret sprawling data representations. The PIVOT operator in SQL.

In SQL Server, strategic use of PIVOT hinges on a deep understanding of the underlying dataset structure and analytical goals. It’s essential to carefully design the column selection and aggregation criteria to ensure meaningful data transformations that align with the queries’ analytic objectives. By leveraging PIVOT, database professionals can empower stakeholders to make data-driven decisions with high precision and clarity, capitalizing on SQL Server’s robust capabilities for sophisticated data manipulation and presentation.

Here the real example a lange table of date

   SELECT top 100 
          [CustomerIdEx] as Customer
          ,[Dt_ym]
          ,[ErpValueNet_sum]
      FROM [OnnShop20REC].[onn].[DBCust_stat_baskets]
      where Dt_ym >= '2023-09-01' order by 1

As we can see, all months and corresponding sales for certain customer is presented in rows.

The PIVOT operator in SQL

If we would like to have months in columns and every customer in rows, then we must apply PIVOT

select 
[CustomerIdEx], 
[2023-09-01],
[2023-10-01],
[2023-11-01],
[2023-12-01],
[2024-01-01],
[2024-02-01],
[2024-03-01],
[2024-04-01],
[2024-05-01],
[2024-06-01],
[2024-07-01],
[2024-08-01],
[2024-09-01],
[2024-10-01] from (
    SELECT top 100 
          [CustomerIdEx] 
          ,[Dt_ym]
          ,[ErpValueNet_sum]
      FROM [OnnShop20REC].[onn].[DBCust_stat_baskets]
      where Dt_ym >= '2023-09-01' order by 1
      ) as p
  pivot
  ( sum([ErpValueNet_sum])
    for [Dt_ym] in (
            [2023-09-01],
            [2023-10-01],
            [2023-11-01],
            [2023-12-01],
            [2024-01-01],
            [2024-02-01],
            [2024-03-01],
            [2024-04-01],
            [2024-05-01],
            [2024-06-01],
            [2024-07-01],
            [2024-08-01],
            [2024-09-01],
            [2024-10-01]   )
  ) as pvt
  order by pvt.[CustomerIdEx]

You can read more about PIVOT here: https://learn.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot?view=sql-server-ver16

We can also build a dynamic sql for similar operations, The PIVOT operator in SQL.

declare @sql_str as nvarchar(max);
declare @key_column as nvarchar(20);
declare @result_columns as nvarchar(max);
declare @source_data as nvarchar(max);
declare @agg_formula as nvarchar(max);
declare @column_field as nvarchar(20);

set @source_data = N'SELECT [Dt_ym]
          ,[CustomerIdEx]
          ,[ErpValueNet_sum]
      FROM [DBCust_stat_baskets]
      where Dt_ym >= ''2023-09-01'' '

set @key_column = '[CustomerIdEx]';
select @result_columns = string_agg(d.[Dt_ym],',') from (
    select [Dt_ym] FROM [DBCust_stat_baskets] where Dt_ym >= '2023-09-01' group by [Dt_ym] ) as d
set @result_columns = '[' + replace(@result_columns,',','],[') +']'
;

set @agg_formula = N'sum([ErpValueNet_sum])';
set @column_field = N'[Dt_ym]'

set @sql_str = 'select ' + @key_column + ',';
set @sql_str = @sql_str + @result_columns + ' from (';
set @sql_str = @sql_str + @source_data + ') as p PIVOT ( ' ;
set @sql_str = @sql_str + @agg_formula + ' for ' + @column_field + ' in ( ' + @result_columns + ' )' 
set @sql_str = @sql_str + ' ) as pvt ' 

EXEC sp_executesql @sql_str;

print @sql_str;

In this example, we’ll write a dynamic SQL (The PIVOT operator in SQL) query to handle scenarios where the pivot columns aren’t known in advance. Dynamic SQL is particularly useful for creating flexible queries that adapt to varying data structures, such as dynamically-generated columns for a pivot operation. Below is a detailed explanation of a dynamic SQL approach to perform a pivot operation in SQL Server.

Dynamic SQL allows for building queries based on dynamic parameters or outputs from other SQL statements. For example, consider a `Sales` table where the `Quarter` values may vary and need to be dynamically determined for the pivot operation. Initially, you’ll retrieve distinct `Quarter` values from the `Sales` table into a string of comma-separated values for use as pivot columns. This can be achieved using the `STUFF` and `FOR XML PATH` techniques to concatenate column names.

sql
DECLARE @columns NVARCHAR(MAX), @sql NVARCHAR(MAX);
-- Dynamically create a string of column names
SELECT @columns = STUFF((SELECT DISTINCT ',' + QUOTENAME(Quarter)
FROM Sales
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 1, '');

-- Construct the dynamic SQL query
SET @sql = N'SELECT Product, ' + @columns + '
FROM (SELECT Product, Quarter, SalesAmount FROM Sales) AS SourceTable
PIVOT (SUM(SalesAmount) FOR Quarter IN (' + @columns + ')) AS PivotTable;';

-- Execute the dynamic SQL query
EXEC sp_executesql @sql;

This code snippet first retrieves unique quarter names dynamically, then builds and executes a pivot query. The PIVOT operator in SQL. It showcases dynamic SQL’s power in handling variable datasets, offering a flexible, adaptable mechanism for data transformation where column values may change over time or differ across environments.

sp_executesql and EXEC

In SQL Server, both `sp_executesql` and `EXEC` can be used to execute dynamic SQL queries, but they have distinct differences and use cases (The PIVOT operator in SQL):

Parameterization

sp_executesql: This is a system stored procedure that allows for parameterized queries. You can define and pass parameters, making the SQL execution more secure and efficient by helping to prevent SQL injection attacks and promoting better query plan reuse. For instance, you can execute a query with variable inputs by defining parameters like this:

DECLARE @sql NVARCHAR(MAX), @paramDefinition NVARCHAR(MAX);
SET @sql = N'SELECT * FROM Sales WHERE Product = @ProductName';
SET @paramDefinition = N'@ProductName NVARCHAR(50)';
EXEC sp_executesql @sql, @paramDefinition, @ProductName = 'A';

EXEC: This command can execute SQL statements or stored procedures but doesn’t inherently support parameterization. It is commonly used for running stored procedures or executing SQL strings directly. If you use it with dynamic SQL, concatenation of strings may be necessary, which can expose the application to SQL injection risks. The PIVOT operator in SQL.

Execution Plan Reuse of sp_executesql and exec

• sp_executesql: Supports better execution plan reuse because it maintains parameterized queries. Since the structure of the query remains consistent across executions with only parameter values changing, SQL Server can efficiently reuse cached execution plans, optimizing performance.
• EXEC: When using dynamic SQL with EXEC, frequent changes in concatenated queries can lead to the creation of multiple execution plans, each tailored to specific string values. This reduces execution plan efficiency and increases resource utilization.

Security and Safety

• sp_executesql: Offers more security advantages by allowing parameterization, which reduces the risk of SQL injection. Parameters are explicitly defined and separated from the query logic.
• EXEC: More prone to SQL injection attacks if not used carefully with input validation and escaping since parameters are often included through string concatenation.

In summary, sp_executesql is generally preferable for executing dynamic SQL as it provides robust support for parameterized queries, enhancing both security and performance through execution plan reuse. On the other hand, EXEC is simpler but less secure and efficient for complex dynamic SQL scenarios. The PIVOT operator in SQL.

The PIVOT operator in SQL

The post The PIVOT operator in SQL appeared first on Customer Experience Management.