The “Marketplace Deduplication” Problem

Context: Large marketplaces (Amazon, eBay, Alibaba, or a niche aggregator) allow thousands of third-party sellers to upload their own product feeds. Product Grouping for E-commerce Datasets indeed.

The Problem:

• Seller A uploads: “Apple iPhone 13, 128GB, Midnight”
• Seller B uploads: “iPhone 13 128 GB Black Unlocked”
• Seller C uploads: “Smartfon Apple iPhone 13 128GB (MLPF3PM/A)”

The “Omnichannel Customer Stitching” Problem (Single Customer View)

Context: A retailer sells through a Website, a Mobile App, and Physical Stores. They want to know if the person browsing the app is the same person buying in the store.

The Problem:

• Record A (Online): email: j.smith@gmail.com, cookie_id: xyz123, behavior: viewed running shoes
• Record B (In-Store POS): card_hash: ****-1234, loyalty_id: 998877, name: John Smith
• Record C (Customer Support): phone: +48 500..., name: Johnny Smith, complaint: "Shoes size 42 too small"

Why it’s hard:

The “Competitor Price Monitoring” Problem

Context: An e-commerce store wants to automatically adjust their prices to be $1 cheaper than their biggest competitor.

The Problem:

• Your Product: “Samsung Galaxy S20 FE 5G Cloud Navy”
• Competitor Site: “Samsung S20 Fan Edition (Navy) – 5G Compatible”

Why it’s hard:

Abstract

• Problem: E-commerce catalogs suffer from redundancy (same product, different sizes/variants).
• Gap: Manual grouping is impossible; Deep Learning is overkill/imprecise for strict SKU grouping.
• Method: We compare Jaccard (Set) vs. TF-IDF (Vector) and propose a normalization pipeline.
• Result: Our method achieved X% accuracy with Y% reduction in computational time.

Set-Theoretic Approaches for Product Grouping (Jaccard)

• Concept: Treats text as a “Bag of Words” (BoW) without weights.
• Key Papers/Concepts:
  • Cohen et al. (2003) often discuss string metrics for entity matching.
  • Shingling / MinHash: In large datasets, calculating Jaccard for all pairs is $O(N^2)$. Literature focuses on Locality Sensitive Hashing (LSH) (MinHash) to approximate Jaccard similarity efficiently.
  • Pros in Literature: High interpretability, excellent for “near-duplicate” detection.
  • Cons: Fails when synonyms are used (e.g., “pants” vs “trousers”) or when word importance varies.

Vector Space Models (TF-IDF)

• Concept: Maps text to a high-dimensional Euclidean space.
• Key Papers/Concepts:
  • Salton et al. (1975) (The foundational VSM paper).
  • Character n-grams: Papers often cite that for noisy user-generated content (UGC), character n-grams outperform word tokens because they handle misspellings morphologically.
  • Pros: Handles “rare words” (like model numbers) better due to IDF (Inverse Document Frequency) weighting.

The State-of-the-Art (Deep Learning / Embeddings)

• If you publish, reviewers will ask: “Why not BERT?”
• SBERT (Sentence-BERT): Current SOTA uses transformer models to generate dense vector embeddings.
• Your Counter-Argument: Deep learning is computationally expensive and “black-box”. For industrial product grouping where exact feature matching (like “Samsung” + “Galaxy”) is critical, classical methods (TF-IDF/Jaccard) often offer better precision and control than semantic embeddings which might group “iPhone 12” with “Samsung S20” because they are both “phones”.

II. Related Work

• Mention LSH (Locality Sensitive Hashing) for Jaccard.
• Mention DBSCAN and Agglomerative Clustering as standard algorithms.
• Cite limitations of BERT in high-precision SKU matching.
• Product Grouping for E-commerce

III. Methodology (The Core)

1. Preprocessing – The Normalization Filter for Product Grouping for E-commerce:
   • Define your Regex rules mathematically.
   • $T_{clean} = f(T_{raw})$ where $f$ removes tokens $t \in \{Dimensions, Colors, Stopwords\}$.
2. Representation:
   • Approach A (Jaccard): $J(A,B) = \frac{|A \cap B|}{|A \cup B|}$
   • Approach B (TF-IDF): Cosine Similarity $\cos(\theta) = \frac{A \cdot B}{||A|| ||B||}$
3. Clustering Algorithm:
   • Explain why you chose Connected Components (Graph theory) for Jaccard or Hierarchical Clustering for TF-IDF.

IV. Experiments (Product Grouping for E-commerce)

• Dataset: Your dataset of “several thousand products”.
• Metrics: You must measure quality.
  • Precision: Are elements in the cluster actually the same product?
  • Recall: Did we find all sizes of that product?
  • F1-Score: Harmonic mean of the two.

V. Results & Discussion

• Hypothesis: Jaccard works better for clean data; TF-IDF works better for noisy data (typos).
• Observation: Jaccard is faster but TF-IDF + N-grams is more robust.

Novel Algorithm:

1. Stage 1 (Blocking): Use Jaccard on tokens to quickly find “Candidate Pairs” (fast, rough filter).
2. Stage 2 (Refinement): Use TF-IDF with Character N-grams on the candidates to calculate a precise similarity score (handles typos).
3. Stage 3 (Decision): Hard threshold (e.g., >0.85).

Relevant Search Terms & Papers

Entity Resolution (ER) is the problem of identifying which records in a database refer to the same real-world entity. An exhaustive ER process involves computing the similarities between pairs of records, which can be very expensive for large datasets. Various blocking techniques can be used to enhance the performance of ER by dividing the records into blocks in multiple ways and only comparing records within the same block.

However, most blocking techniques process blocks separately and do not exploit the results of other blocks. In this paper (https://dl.acm.org/doi/pdf/10.1145/1559845.1559870), authors propose an iterative blocking framework where the ER results of blocks are reflected to subsequently processed blocks. Blocks are now iteratively processed until no block contains any more matching records.

Compared to simple blocking, iterative blocking may achieve higher accuracy because reflecting the ER results of blocks to other blocks may generate additional record matches. Iterative blocking may also be more efficient because processing a block now saves the processing time for other blocks.

• “Short Text Clustering for E-commerce”
• “Product Entity Resolution with Noise”
• “Comparison of Jaccard and Cosine Similarity in Text Mining”
• “Blocking techniques for Entity Resolution” (This is crucial for scaling to thousands/millions of products). Product Grouping for E-commerce.

Specific types of papers to look for:

1. Koporec et al.: Papers on combining Jaccard with other metrics. https://www.sciencedirect.com/science/article/pii/S1364815225002981
2. Ganesan et al.: Research on “abstractive summarization” or short-text clustering in retail.

FAQ: Product Grouping in Noisy E-Commerce Datasets

The post Product Grouping in Noisy E-commerce Datasets appeared first on Customer Experience Management.

-- The query
SELECT 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1) order_date
from ss
group by 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1)

-- The query
SELECT 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1) order_date
from ss
group by 
   ss.SOLDTONBR,
   datefromparts(year(ORDERDATE),month(ORDERDATE),1)

How to optimize this query?

The Problem of optimizing SQL query: Preventing Index Usage

The query is not SARGable (Searchable Argument-able). This means the database engine can’t use an index to find the data it needs directly. Instead, it must perform a full table scan, reading all billion rows from the disk, calculating the first day of the month for each one, and then grouping the results. This is incredibly slow and resource-intensive.

Optimization Strategies

Create a Composite Index – The Biggest Win

Regardless of how you rewrite the query, the most important step is to have the right index. Since you are grouping by SOLDTONBR and then by a derivative of ORDERDATE, the ideal index would cover both columns.

CREATE INDEX IX_ss_SoldToNbr_OrderDate ON ss (SOLDTONBR, ORDERDATE);

This index allows the database to quickly access the data pre-sorted by SOLDTONBR and ORDERDATE, making the grouping operation vastly more efficient. Optimizing SQL query.

Optimisation tips you can also find here: 15 Essential SQL Tips You Can’t Live Without

Column store indices for MS SQL Server you can find here

Use More Efficient Date Functions

Instead of combining YEAR and MONTH, use a single, more efficient function to truncate the date to the beginning of the month. The best function depends on your specific database system. These functions are often better optimized internally. Optimizing SQL query.

• SQL Server:SQL-- Best for modern SQL Server SELECT ss.SOLDTONBR, DATETRUNC(month, ORDERDATE) AS order_date FROM ss GROUP BY ss.SOLDTONBR, DATETRUNC(month, ORDERDATE);
• PostgreSQL / Redshift:SQLSELECT ss.SOLDTONBR, DATE_TRUNC('month', ORDERDATE) AS order_date FROM ss GROUP BY ss.SOLDTONBR, DATE_TRUNC('month', ORDERDATE);
• MySQL:SQLSELECT ss.SOLDTONBR, DATE_FORMAT(ORDERDATE, '%Y-%m-01') AS order_date FROM ss GROUP BY ss.SOLDTONBR, order_date; -- MySQL allows using alias in GROUP BY
• Oracle:SQLSELECT ss.SOLDTONBR, TRUNC(ORDERDATE, 'MM') AS order_date FROM ss GROUP BY ss.SOLDTONBR, TRUNC(ORDERDATE, 'MM');

While better, these still apply a function to the column, which can limit performance. The index from step 1 is still what makes the biggest difference.

3. The Gold Standard: A Date Dimension Table

For very large tables (like yours), the best practice is to use a date dimension table. This is a static lookup table containing one row for every day over a long period (e.g., 100 years). It has pre-calculated columns for everything you might need: first day of the month, week of the year, quarter, etc.

Here you can see how to create a Date Dimension Table using recursive CTE: How Works Recursive CTE in SQL Server?

SET DATEFIRST  1,
    DATEFORMAT ymd,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20220101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 5, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,

        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
      
)
SELECT
    dd.*,
    k.WeekOfYear_fdt
FROM
    DateDimension dd
LEFT JOIN
    (
        SELECT
            Year,
            WeekOfYear,
            MIN(Date) as WeekOfYear_fdt
        FROM DateDimension
        GROUP BY Year, WeekOfYear
    ) AS k ON k.Year = dd.Year AND k.WeekOfYear = dd.WeekOfYear
ORDER BY
    Date
OPTION (MAXRECURSION 0);

SET DATEFIRST  1,
    DATEFORMAT ymd,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20220101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 5, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,

        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
      
)
SELECT
    dd.*,
    k.WeekOfYear_fdt
FROM
    DateDimension dd
LEFT JOIN
    (
        SELECT
            Year,
            WeekOfYear,
            MIN(Date) as WeekOfYear_fdt
        FROM DateDimension
        GROUP BY Year, WeekOfYear
    ) AS k ON k.Year = dd.Year AND k.WeekOfYear = dd.WeekOfYear
ORDER BY
    Date
OPTION (MAXRECURSION 0);

Example DimDate Table:

How to Use It:

You join your main table to DimDate. The expensive GROUP BY calculation is replaced with a fast join and a group on a pre-calculated column.

SQL. Optimizing SQL query.

SELECT
    ss.SOLDTONBR,
    d.FirstOfMonth AS order_date
FROM
    ss
INNER JOIN
    DimDate d ON ss.ORDERDATE = d.FullDate -- Assumes ORDERDATE has no time component
GROUP BY
    ss.SOLDTONBR,
    d.FirstOfMonth;

Why this is the most efficient:

• No on-the-fly calculations: The grouping is done on a column that is already calculated (d.FirstOfMonth).
• Index-friendly: The join ss.ORDERDATE = d.FullDate can use indexes on both columns for maximum speed.
• Flexibility: It makes grouping by any other time period (week, quarter) trivial and just as fast.

Grouping by Week of the Year

To group by week, you can apply the same principles.

1. Function-based (Less Efficient):Use a function like DATETRUNC(‘week’, ORDERDATE) or DATEPART(week, ORDERDATE).SQL-- Example for PostgreSQL / SQL Server SELECT ss.SOLDTONBR, DATETRUNC(week, ORDERDATE) as order_week FROM ss GROUP BY ss.SOLDTONBR, DATETRUNC(week, ORDERDATE);
2. Date Dimension Table (Most Efficient):This is the ideal way. Just change the column in your GROUP BY clause.SQLSELECT ss.SOLDTONBR, d.WeekOfYear, -- Or a column representing the first day of the week d.Year FROM ss INNER JOIN DimDate d ON ss.ORDERDATE = d.FullDate GROUP BY ss.SOLDTONBR, d.WeekOfYear, d.Year;

Summary of Recommendations

For a table with billions of records, follow these steps for maximum performance (Optimizing SQL query.):

1. Immediately create a composite index: CREATE INDEX IX_ss_SoldToNbr_OrderDate ON ss (SOLDTONBR, ORDERDATE);
2. For the ultimate solution, create and populate a Date Dimension table. Join to this table to perform your aggregations. This is the most scalable and flexible method.
3. If you cannot create a dimension table, replace your DATEFROMPARTS(YEAR(...), MONTH(...)) logic with a more efficient, platform-specific function like DATETRUNC(). Optimizing SQL query.

The post Optimizing SQL query for billions of records table – a powerful tips appeared first on Customer Experience Management.

SQL is the backbone of data-driven decision-making, and mastering it can dramatically improve how you interact with databases. Whether you’re a seasoned developer or just starting out, writing efficient, readable, and scalable SQL queries is a skill that pays off daily. In this post, I’ve compiled ten essential tips that will help you write smarter SQL—tips that I’ve learned, refined, and shared over time. These aren’t just theoretical best practices; they’re practical techniques that can make your queries faster, your code cleaner, and your debugging easier.

You can try Microsoft SQL Server from here: https://www.microsoft.com/pl-pl/sql-server/sql-server-downloads. And developer edition is here

From avoiding SELECT * to choosing the right join types, each tip is designed to help you think critically about how your queries perform and how they scale. You’ll also learn how to use indexes effectively, filter data early, and make smart choices between EXISTS and IN. Each section includes a short summary and a link to a full post where you can dive deeper into the topic. Whether you’re optimizing a legacy system or building something new, these tips will help you get the most out of SQL—and avoid common pitfalls that slow down your work.

Select Only What You Need, SQL Tips no 1

Avoid SELECT * and specify only the columns you need. This reduces data transfer, memory usage, and improves query speed. For example, instead of pulling all employee data, just select employee_id, first_name, and last_name. Read more

Optimize Join Conditions for Performance

Avoid non-SARGable joins that prevent index usage. Instead of applying functions to columns in join conditions, restructure the logic to preserve index efficiency. This dramatically improves query speed. Read the full guide

Use the PIVOT Operator for Better Reporting

Transform row-based data into columnar format using PIVOT. This is ideal for cross-tab reports and trend analysis, especially when comparing metrics across time or categories. Explore the PIVOT tutorial

Master Recursive CTEs for Hierarchical Data

Recursive Common Table Expressions (CTEs) allow you to elegantly query hierarchical or tree-structured data. They’re powerful for tasks like organizational charts or category trees. Learn about recursive CTEs

Set the First Day of the Week with DATEFIRST

Use SET DATEFIRST to control how SQL Server interprets weekday numbers. This is crucial for accurate time-based reporting and week-based aggregations. See how to use DATEFIRST

Update Multiple Tables with Conditions

Learn how to structure multi-table updates using joins and conditional logic. This technique is essential for synchronizing data across related tables. Read the multi-table update example

Filter Early with WHERE Clauses

Apply filters as early as possible to reduce the number of rows processed in joins and aggregations. Optimize your filtering

Use UNION ALL Instead of UNION

UNION ALL is faster than UNION because it skips duplicate elimination. Use it when duplicates aren’t a concern. Performance tip explained

Avoid Functions on Indexed Columns

Using functions like LOWER() or DATEADD() on indexed columns disables index usage. Rewrite conditions to preserve index paths. Join optimization example

Explore SQL for Business Analytics

Advanced SQL techniques like statistical analysis, predictive modeling, and time series forecasting are essential for business analysts. Learn how to combine SQL with Python for deeper insights. Check out the full course

5 additional SQL tips

Use CTEs for Readability and Reuse

Common Table Expressions (CTEs) make complex queries easier to read and maintain. They allow you to define temporary result sets that can be referenced multiple times. SQL Tips

WITH recentorders AS (
  SELECT customerid, orderdate
  FROM orders
  WHERE orderdate > CURRENTDATE - INTERVAL '30 days'
)
SELECT customerid, COUNT(*) AS ordercount
FROM recentorders
GROUP BY customer_id;

Avoid Functions on Indexed Columns

Using functions on indexed columns disables index usage, slowing down queries. Instead, transform the value before comparison. SQL Tips

-- Avoid
SELECT FROM users WHERE LOWER(email) = 'test@example.com';
-- Better
SELECT FROM users WHERE email = 'test@example.com';

Use CASE for Conditional Logic

CASE lets you embed conditional logic directly in your queries, useful for categorizing or transforming data.

SELECT name,
  CASE
    WHEN score >= 90 THEN 'Excellent'
    WHEN score >= 75 THEN 'Good'
    ELSE 'Needs Improvement'
  END AS performance
FROM students;

Optimize Aggregations with GROUP BY

When aggregating data, ensure you’re grouping only necessary columns to avoid performance hits and incorrect results.

SELECT department, AVG(salary) AS avg_salary
FROM employees
GROUP BY department;

Use Parameterized Queries to Prevent SQL Injection

Always use parameterized queries in application code to protect against SQL injection.

-- Example in Python with psycopg2
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))

Remember – tips summary

Use Joins Efficiently

Choose the right join type—INNER JOIN for matched rows, and avoid CROSS JOIN unless necessary. Efficient joins reduce unnecessary data processing and improve clarity.

Filter Data Early

Apply filters as soon as possible in your query using WHERE clauses. This minimizes the number of rows processed in joins and aggregations, leading to faster execution. SQL Tips no 3

Use Indexes Wisely

Indexes speed up data retrieval, especially in WHERE, JOIN, and ORDER BY clauses. But don’t over-index—too many can slow down INSERT, UPDATE, and DELETE operations.

Avoid Subqueries in WHERE Clauses

Correlated subqueries can be slow. Replace them with joins when possible to improve performance and readability. SQL Tips no 4

Use UNION ALL Instead of UNION

UNION removes duplicates, which is costly. If duplicates aren’t a concern, use UNION ALL for faster results.

Limit Your Results

Use LIMIT or TOP to restrict the number of rows returned. This is especially useful for pagination or sampling large datasets.

Be Cautious with LIKE and Functions

Avoid leading wildcards in LIKE and functions in WHERE clauses—they prevent index usage. Instead, use indexed columns and consistent casing.

Use EXISTS Instead of IN

EXISTS is often faster than IN because it stops scanning once a match is found. Use it for subqueries checking row existence.

Use Appropriate Data Types

Choosing the right data type—like TINYINT over INT or CHAR over VARCHAR—can save space and improve performance.

SQL Server Management Studio

SSMS as a Comprehensive SQL Environment
SQL Server Management Studio (https://learn.microsoft.com/en-us/ssms/) is a powerful, integrated environment for managing SQL Server infrastructure. It provides tools for writing, executing, and debugging SQL queries, as well as managing databases, tables, views, and stored procedures. SSMS supports both on-premises and cloud-based SQL Server instances, making it versatile for hybrid environments. Its intuitive interface includes Object Explorer for navigating server components and Query Editor for crafting and testing SQL scripts. Whether you’re a database administrator or developer, SSMS offers a unified workspace that streamlines daily tasks and enhances productivity through built-in templates, syntax highlighting, and error diagnostics.

Security, Performance, and Monitoring Tools
SSMS includes robust features for security management, such as configuring roles, permissions, and auditing access. It also provides performance tuning tools like the Database Engine Tuning Advisor and graphical execution plans to help identify bottlenecks. With Activity Monitor, users can track real-time server performance, view active sessions, and analyze resource usage. These tools empower teams to maintain optimal database health and ensure compliance with organizational policies. SSMS also integrates with SQL Server Agent for scheduling jobs and alerts, making it a central hub for automation and proactive monitoring across enterprise environments.

Integration, Extensibility, and Collaboration
SSMS supports integration with source control systems like Git, enabling versioning and collaborative development. It allows exporting and importing data via wizards, scripting database objects, and generating reports for documentation. Users can extend SSMS functionality through add-ins and connect to Azure services for cloud-based analytics and storage. Its support for multiple query windows and tabbed editing enhances multitasking, while customizable keyboard shortcuts and themes improve user experience. SSMS continues to evolve with regular updates, ensuring compatibility with the latest SQL Server features and providing a stable platform for modern data operations.

The post 15 Essential SQL Tips You Can’t Live Without appeared first on Customer Experience Management.

The query to be optimized

Here there is a query. The table has no index. How can we optimize the query and the table reading performance setting up indices? select k.*, 

select k.*, 
       c.[Top_segment]
      ,c.[CompanyType]
      ,c.[Potencial] from (
    select 
          a.[Dt_ym]
          ,a.[CustomerIdEx]
          ,sum(a.ErpRegisterDoc_count) ErpRegisterDoc_count
          ,sum(a.ErpValueNet_sum) ErpValueNet_sum, sum(b.ErpValueNet_sum) ErpValueNet_sum_py
    FROM [onn].[DBCust_stat_baskets] a with (nolock)
      left join [onn].[DBCust_stat_baskets] b with (nolock) 
          on dateadd(year, 1, b.dt_ym) = a.Dt_ym and b.[CustomerIdEx] = a.CustomerIdEx
group by           a.[Dt_ym]
          ,a.[CustomerIdEx]
 ) as k 
  left join [onn].[DBCust_erp] c with (nolock) on k.CustomerIdEx = c.Cust_nr

select k.*, 
       c.[Top_segment]
      ,c.[CompanyType]
      ,c.[Potencial] from (
    select 
          a.[Dt_ym]
          ,a.[CustomerIdEx]
          ,sum(a.ErpRegisterDoc_count) ErpRegisterDoc_count
          ,sum(a.ErpValueNet_sum) ErpValueNet_sum, sum(b.ErpValueNet_sum) ErpValueNet_sum_py
    FROM [onn].[DBCust_stat_baskets] a with (nolock)
      left join [onn].[DBCust_stat_baskets] b with (nolock) 
          on dateadd(year, 1, b.dt_ym) = a.Dt_ym and b.[CustomerIdEx] = a.CustomerIdEx
group by           a.[Dt_ym]
          ,a.[CustomerIdEx]
 ) as k 
  left join [onn].[DBCust_erp] c with (nolock) on k.CustomerIdEx = c.Cust_nr

The biggest performance issues here are the complete lack of indexes and a join condition that prevents the use of an index (even if one existed). You can dramatically speed this up by creating the right indexes and making a small but critical change to the query.

Setting Up the Right Indices

An index acts like a book’s index, allowing SQL Server to find data quickly without scanning the entire table. Based on your query’s JOIN and GROUP BY clauses, these are the indexes you need.

On the DBCust_stat_baskets table:

This table is joined to itself and grouped by CustomerIdEx and Dt_ym. A covering index will be most effective here. This special type of index not only includes the key columns for searching but also the data columns the query needs, allowing the entire subquery to be satisfied from the index alone.

SQL

CREATE NONCLUSTERED INDEX IX_DBCust_stat_baskets_Customer_Date
ON [onn].[DBCust_stat_baskets] ([CustomerIdEx], [Dt_ym])
INCLUDE ([ErpRegisterDoc_count], [ErpValueNet_sum]);

• ON ([CustomerIdEx], [Dt_ym]): This covers your JOIN and GROUP BY columns. The order is important; CustomerIdEx is more selective, so it comes first.
• INCLUDE (...): This “covers” the SUM calculations, so SQL Server doesn’t have to look up the data in the main table, which is a massive performance boost.

On the DBCust_erp table:

This table is joined on the Cust_nr column. It also needs an index to make the final LEFT JOIN fast.

SQL

CREATE NONCLUSTERED INDEX IX_DBCust_erp_Cust_nr
ON [onn].[DBCust_erp] ([Cust_nr])
INCLUDE ([Top_segment], [CompanyType], [Potencial]);

• ON ([Cust_nr]): Speeds up the join to the subquery k.
• INCLUDE (...): Covers the columns you are selecting, again avoiding a second lookup to the main table.

Optimizing the T-SQL Query 🚀

Your query has a “non-SARGable” join condition. SARGable means Search Argumentable, allowing SQL to use an index.

The problem is this line: dateadd(year, 1, b.dt_ym) = a.Dt_ym. By applying the dateadd function to the column b.dt_ym, you force SQL Server to calculate a new value for every single row in the table before it can perform the join. This guarantees a slow table scan.

The fix is to apply the function to the other side of the equation, so the column itself remains “clean.”

Change this: ON dateadd(year, 1, b.dt_ym) = a.Dt_ym

To this: ON b.dt_ym = dateadd(year, -1, a.Dt_ym)

This small change allows SQL Server to take a value from a, calculate the prior year’s date once, and then perform a highly efficient index seek on b.dt_ym to find the matching row.

The Final Optimized Query

For better readability, I’ve rewritten your query using a Common Table Expression (CTE). The logic is the same, but it’s easier to follow. This version incorporates the SARGable join fix.

SQL

-- Use a CTE for better readability to get the current and prior year sums
WITH CustomerMonthlyStats AS (
    SELECT
        a.[Dt_ym],
        a.[CustomerIdEx],
        SUM(a.ErpRegisterDoc_count) AS ErpRegisterDoc_count,
        SUM(a.ErpValueNet_sum) AS ErpValueNet_sum,
        SUM(b.ErpValueNet_sum) AS ErpValueNet_sum_py -- Sum from the previous year
    FROM
        [onn].[DBCust_stat_baskets] AS a
    LEFT JOIN
       [onn].[DBCust_stat_baskets] AS b
        -- OPTIMIZED: The function is now on the outer side of the join, making it SARGable
        ON a.[CustomerIdEx] = b.[CustomerIdEx]
        AND b.dt_ym = DATEADD(year, -1, a.Dt_ym)
    GROUP BY
        a.[Dt_ym],
        a.[CustomerIdEx]
)
-- Final SELECT joining the aggregated stats with customer details
SELECT
    k.*,
    c.[Top_segment],
    c.[CompanyType],
    c.[Potencial]
FROM
    CustomerMonthlyStats AS k
LEFT JOIN
    [onn].[DBCust_erp] AS c ON k.CustomerIdEx = c.Cust_nr;

By creating these indexes and applying this simple query fix, you should see a dramatic improvement in performance, going from potentially minutes to seconds. I’d also recommend removing the WITH (NOLOCK) hints after these changes, as they won’t be necessary and can lead to reading “dirty” or incorrect data.

You also converted my query to CTE, but I cant use cte in Power BI direct query, that is why I used like that. Do I don’t know something or CTE is really better then my version of “cascade” querying?

That’s an excellent point about Power BI DirectQuery’s limitations with CTEs. You are correct; you often have to use subqueries (also called derived tables) instead.

To answer your main question: for this specific query, a CTE is not better than your subquery in terms of performance. The SQL Server query optimizer is smart enough to see that both your subquery version and the CTE version are logically identical. It will generate the exact same execution plan for both, meaning they will run at the same speed.

The primary advantage of a CTE is readability and maintainability, not performance. It helps break down complex logic into clean, sequential steps, making the query easier for a human to understand and debug later.

The Important Part: The Optimization Itself

The crucial takeaway from the previous answer was not the switch to a CTE, but these two points:

1. Creating the right indexes on the tables.
2. Fixing the non-SARGable join condition.

As long as you apply those two changes, your original query structure is perfectly fine and will be just as fast as the CTE version.

Your Original Query, Optimized

Here is your original query with the single, critical change to the JOIN condition. This version avoids CTEs and is ready for Power BI DirectQuery, while still containing the essential performance fix.

SQL

SELECT
    k.*,
    c.[Top_segment],
    c.[CompanyType],
    c.[Potencial]
FROM (
    -- This subquery is functionally the same as the CTE
    SELECT
        a.[Dt_ym],
        a.[CustomerIdEx],
        SUM(a.ErpRegisterDoc_count) AS ErpRegisterDoc_count,
        SUM(a.ErpValueNet_sum) AS ErpValueNet_sum,
        SUM(b.ErpValueNet_sum) AS ErpValueNet_sum_py
    FROM
        [onn].[DBCust_stat_baskets] AS a WITH (NOLOCK)
    LEFT JOIN
        [onn].[DBCust_stat_baskets] AS b WITH (NOLOCK)
        -- The single most important optimization is right here:
        ON a.CustomerIdEx = b.CustomerIdEx AND b.dt_ym = DATEADD(year, -1, a.Dt_ym)
    GROUP BY
        a.[Dt_ym],
        a.[CustomerIdEx]
) AS k
LEFT JOIN
   [onn].[DBCust_erp] AS c WITH (NOLOCK) ON k.CustomerIdEx = c.Cust_nr;

The post Query optimization with join condition appeared first on Customer Experience Management.

A recursive CTE in SQL Server consists of:

1. Anchor member – the starting point.
2. Recursive member – repeatedly executed until a condition is met.

Recursive CTE in SQL Server
-- Prevent regional settings from interfering with date interpretation
SET DATEFIRST  7,         -- Set Sunday as the first day of the week
    DATEFORMAT mdy,       -- Set date format to Month-Day-Year
    LANGUAGE   US_ENGLISH;-- Use US English for language settings

-- Declare start and cutoff dates
DECLARE @StartDate  date = '20100101'; -- Start date: Jan 1, 2010
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));
-- Cutoff date: one day before Jan 1, 2013 (i.e., Dec 31, 2012)

-- Recursive CTE to generate a sequence of numbers from 0 to number of days between StartDate and CutoffDate
WITH seq(n) AS 
(
    SELECT 0               -- Anchor member: start with 0
    UNION ALL
    SELECT n + 1           -- Recursive member: increment n
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
    -- Stop when n reaches the number of days between StartDate and CutoffDate
)
SELECT n FROM seq          -- Select the generated sequence
ORDER BY n                 -- Order the result
OPTION (MAXRECURSION 0);   -- Allow unlimited recursion (default is 100)

Recursive CTE in SQL Server
-- Prevent regional settings from interfering with date interpretation
SET DATEFIRST  7,         -- Set Sunday as the first day of the week
    DATEFORMAT mdy,       -- Set date format to Month-Day-Year
    LANGUAGE   US_ENGLISH;-- Use US English for language settings

-- Declare start and cutoff dates
DECLARE @StartDate  date = '20100101'; -- Start date: Jan 1, 2010
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));
-- Cutoff date: one day before Jan 1, 2013 (i.e., Dec 31, 2012)

-- Recursive CTE to generate a sequence of numbers from 0 to number of days between StartDate and CutoffDate
WITH seq(n) AS 
(
    SELECT 0               -- Anchor member: start with 0
    UNION ALL
    SELECT n + 1           -- Recursive member: increment n
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
    -- Stop when n reaches the number of days between StartDate and CutoffDate
)
SELECT n FROM seq          -- Select the generated sequence
ORDER BY n                 -- Order the result
OPTION (MAXRECURSION 0);   -- Allow unlimited recursion (default is 100)

Recursive CTE in SQL Server.

Data table for BI

In business intelligence (BI) and data warehousing, a date dimension table is a foundational component that enables powerful time-based analysis. This SQL script demonstrates how to dynamically generate a comprehensive date dimension using a recursive Common Table Expression (CTE). Starting from a defined StartDate, it builds a sequence of dates up to a CutoffDate, enriching each date with attributes such as day name, month, quarter, week number, and weekend flag. This structure supports flexible filtering, grouping, and visualization in BI tools like Power BI, Tableau, or SQL Server Reporting Services (SSRS), and ensures consistent date logic across reports and dashboards.

Recursive CTE in SQL Server

1. Translate the generated numbers into actual dates.
2. Create a date dimension table suitable for use in BI tools like Power BI or Tableau.

Recursive CTE in SQL Server
-- Regional settings
SET DATEFIRST  7,
    DATEFORMAT mdy,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20100101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,
        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
)
SELECT * FROM DateDimension
ORDER BY Date
OPTION (MAXRECURSION 0);

Recursive CTE in SQL Server
-- Regional settings
SET DATEFIRST  7,
    DATEFORMAT mdy,
    LANGUAGE   US_ENGLISH;

-- Define date range
DECLARE @StartDate  date = '20100101';
DECLARE @CutoffDate date = DATEADD(DAY, -1, DATEADD(YEAR, 3, @StartDate));

-- Recursive CTE to generate date dimension
WITH seq(n) AS 
(
    SELECT 0
    UNION ALL
    SELECT n + 1
    FROM seq
    WHERE n < DATEDIFF(DAY, @StartDate, @CutoffDate)
),
DateDimension AS
(
    SELECT 
        DATEADD(DAY, n, @StartDate) AS Date,
        DATENAME(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayName,
        DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) AS DayOfWeek,
        DATEPART(DAY, DATEADD(DAY, n, @StartDate)) AS Day,
        DATEPART(MONTH, DATEADD(DAY, n, @StartDate)) AS Month,
        DATENAME(MONTH, DATEADD(DAY, n, @StartDate)) AS MonthName,
        DATEPART(YEAR, DATEADD(DAY, n, @StartDate)) AS Year,
        DATEPART(QUARTER, DATEADD(DAY, n, @StartDate)) AS Quarter,
        DATEPART(WEEK, DATEADD(DAY, n, @StartDate)) AS WeekOfYear,
        CASE 
            WHEN DATEPART(WEEKDAY, DATEADD(DAY, n, @StartDate)) IN (1, 7) THEN 1
            ELSE 0
        END AS IsWeekend
    FROM seq
)
SELECT * FROM DateDimension
ORDER BY Date
OPTION (MAXRECURSION 0);

Trad more here and How To Set The First Day of Week DATEFIRST

Next Steps

• Add fiscal year columns
• Add holiday flags (if you provide a list)
• Localize day/month names to Polish if needed

Fiscal Year Columns

Recursive CTE in SQL Server
-- Assuming fiscal year starts in April
SELECT
    Date,
    CASE 
        WHEN MONTH(Date) >= 4 THEN YEAR(Date)
        ELSE YEAR(Date) - 1
    END AS FiscalYear,
    CASE 
        WHEN MONTH(Date) BETWEEN 4 AND 6 THEN 1
        WHEN MONTH(Date) BETWEEN 7 AND 9 THEN 2
        WHEN MONTH(Date) BETWEEN 10 AND 12 THEN 3
        ELSE 4
    END AS FiscalQuarter
FROM DateDimension

Recursive CTE in SQL Server
-- Assuming fiscal year starts in April
SELECT
    Date,
    CASE 
        WHEN MONTH(Date) >= 4 THEN YEAR(Date)
        ELSE YEAR(Date) - 1
    END AS FiscalYear,
    CASE 
        WHEN MONTH(Date) BETWEEN 4 AND 6 THEN 1
        WHEN MONTH(Date) BETWEEN 7 AND 9 THEN 2
        WHEN MONTH(Date) BETWEEN 10 AND 12 THEN 3
        ELSE 4
    END AS FiscalQuarter
FROM DateDimension

Holiday Flags

It requires a holiday list. Assuming you have a table called HolidayList with a column HolidayDate.

Recursive CTE in SQL Server
SELECT
    d.*,
    CASE 
        WHEN h.HolidayDate IS NOT NULL THEN 1
        ELSE 0
    END AS IsHoliday
FROM DateDimension d
LEFT JOIN HolidayList h ON d.Date = h.HolidayDate

Recursive CTE in SQL Server
SELECT
    d.*,
    CASE 
        WHEN h.HolidayDate IS NOT NULL THEN 1
        ELSE 0
    END AS IsHoliday
FROM DateDimension d
LEFT JOIN HolidayList h ON d.Date = h.HolidayDate

Localize Day/Month Names to Polish

SELECT
    Date,
    CASE DATENAME(WEEKDAY, Date)
        WHEN 'Monday' THEN 'Poniedziałek'
        WHEN 'Tuesday' THEN 'Wtorek'
        WHEN 'Wednesday' THEN 'Środa'
        WHEN 'Thursday' THEN 'Czwartek'
        WHEN 'Friday' THEN 'Piątek'
        WHEN 'Saturday' THEN 'Sobota'
        WHEN 'Sunday' THEN 'Niedziela'
    END AS DayNamePL,
    CASE DATENAME(MONTH, Date)
        WHEN 'January' THEN 'Styczeń'
        WHEN 'February' THEN 'Luty'
        WHEN 'March' THEN 'Marzec'
        WHEN 'April' THEN 'Kwiecień'
        WHEN 'May' THEN 'Maj'
        WHEN 'June' THEN 'Czerwiec'
        WHEN 'July' THEN 'Lipiec'
        WHEN 'August' THEN 'Sierpień'
        WHEN 'September' THEN 'Wrzesień'
        WHEN 'October' THEN 'Październik'
        WHEN 'November' THEN 'Listopad'
        WHEN 'December' THEN 'Grudzień'
    END AS MonthNamePL
FROM DateDimension

SELECT
    Date,
    CASE DATENAME(WEEKDAY, Date)
        WHEN 'Monday' THEN 'Poniedziałek'
        WHEN 'Tuesday' THEN 'Wtorek'
        WHEN 'Wednesday' THEN 'Środa'
        WHEN 'Thursday' THEN 'Czwartek'
        WHEN 'Friday' THEN 'Piątek'
        WHEN 'Saturday' THEN 'Sobota'
        WHEN 'Sunday' THEN 'Niedziela'
    END AS DayNamePL,
    CASE DATENAME(MONTH, Date)
        WHEN 'January' THEN 'Styczeń'
        WHEN 'February' THEN 'Luty'
        WHEN 'March' THEN 'Marzec'
        WHEN 'April' THEN 'Kwiecień'
        WHEN 'May' THEN 'Maj'
        WHEN 'June' THEN 'Czerwiec'
        WHEN 'July' THEN 'Lipiec'
        WHEN 'August' THEN 'Sierpień'
        WHEN 'September' THEN 'Wrzesień'
        WHEN 'October' THEN 'Październik'
        WHEN 'November' THEN 'Listopad'
        WHEN 'December' THEN 'Grudzień'
    END AS MonthNamePL
FROM DateDimension

This extended SQL script builds a robust date dimension table tailored for business intelligence applications. Starting with a recursive CTE to generate a sequence of dates, it enriches each entry with key calendar attributes such as day names, month names, quarters, and weekend flags. Additional enhancements include fiscal year logic, holiday indicators (based on a provided list), and localization of day and month names to Polish. These features ensure the table supports flexible, localized, and context-aware time-based analysis in tools like Power BI, Tableau, or SQL Server Reporting Services.

The post How Works Recursive CTE in SQL Server? appeared first on Customer Experience Management.

Vector vs lexical search

An easy way to introduce vector search is by comparing it to the more conventional lexical search that you’re probably used to. Vector search, also commonly known as semantic search, and lexical search work very differently.

Lexical search is the kind of search that we’ve all been using for years. To summarize it very briefly, it doesn’t try to understand the real meaning of what is indexed and queried, instead, it makes a big effort to lexically match the literals of the words or variants of them like stemming words, or synonyms, etc.. That makes what the user types in a query with all the literals that have been previously indexed into the database. The similarity is replaced by ranking algorithm, such as TF-IDF.

Documents are tokenized and analyzed. Then, the resulting terms are indexed in an inverted index, which simply maps the analyzed terms to the documents containing them. Searching for “yellow texas roses” will match all documents with varying scores.

Semantic search – the whole purpose of semantic search is to index data in such a way that it can be searched based on the meaning it represents.

What’s the difference between semantic search and lexical search?

Lexical search doesn’t try to understand the real meaning of what is indexed and queried- it matches the literals of the words or their variants. In contrast, vector search indexes data in a way that allows it to be searched based on the meaning it represents.

Read more about vector similarity https://www.elastic.co/search-labs/blog/introduction-to-vector-search

Semantic search with elasticsearch

Load model. See details here and here.

from elasticsearch import Elasticsearch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-base")
VECTOR_DIMENSION = model.get_sentence_embedding_dimension()
print(f"Model loaded successfully.VECTOR_DIMENSION {VECTOR_DIMENSION}")

from elasticsearch import Elasticsearch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-base")
VECTOR_DIMENSION = model.get_sentence_embedding_dimension()
print(f"Model loaded successfully.VECTOR_DIMENSION {VECTOR_DIMENSION}")

Start elastic server

from elasticsearch import Elasticsearch
ES_HOST = "http://localhost:9000"
es = Elasticsearch(hosts=[ES_HOST])
print(f"Connection successful: {es.info().body['cluster_name']}")

from elasticsearch import Elasticsearch
ES_HOST = "http://localhost:9000"
es = Elasticsearch(hosts=[ES_HOST])
print(f"Connection successful: {es.info().body['cluster_name']}")

Index product data to elastic database

# Fetch products from database
def select_products():
    q = """
        SELECT *
        FROM [DB_Products] 
        where [BrandName] in ('PCE','De Walt')
        """
    dfp = read_from_sql_server(q, odbc_conect='DSN=SQLxx')
    return dfp

# >-------------------------------------------------
df_docs = select_products()
print(df_docs.info())
# -------------------------------------------------

# Fetch products from database
def select_products():
    q = """
        SELECT *
        FROM [DB_Products] 
        where [BrandName] in ('PCE','De Walt')
        """
    dfp = read_from_sql_server(q, odbc_conect='DSN=SQLxx')
    return dfp

# >-------------------------------------------------
df_docs = select_products()
print(df_docs.info())
# -------------------------------------------------

# Index product names into elastic
index_mapping = {
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims":  VECTOR_DIMENSION,
                "index": True,
                "similarity": "cosine"
                },

            "product_name": { 
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256 
                    }
                  }
                },
            "product_name_org": {
                "type": "keyword",
                "ignore_above": 256
                },
            "prodidx": {
                "type": "keyword",
                "ignore_above": 256
                },
            "img_url": {
                "type": "keyword",
                "ignore_above": 512 
                },
            }
        }

# --- 4. Create the Index -------------------------------------------------

INDEX_NAME = 'prod_names_for_search_hybrid'

import time
es.options(ignore_status=[400, 404]).indices.delete(index=INDEX_NAME, ignore_unavailable=True)
time.sleep(3) # Give a moment for index deletion to propagate

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, mappings=index_mapping)
    print("Index created.")
else:
    print("Index already exists.")
# -----------------------------------------------------------------------

# Indexing product data ------------------------------------------------- 
product_names = df_docs['ProductName'].apply(lambda x: x.lower()[:256]).to_list()

i = 1
for name in product_names:
    print(f"Indexing:{i} {name}")
    i += 1
    
    vector = model.encode(f"passage: {name}").tolist()
    doc = {
        "Product_name": name,
        "ProductNameVector": vector
    }
    
    es.index(index=INDEX_NAME, document=doc, refresh=True)

# ---------------------------------------------------------------------

# Index product names into elastic
index_mapping = {
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims":  VECTOR_DIMENSION,
                "index": True,
                "similarity": "cosine"
                },

            "product_name": { 
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256 
                    }
                  }
                },
            "product_name_org": {
                "type": "keyword",
                "ignore_above": 256
                },
            "prodidx": {
                "type": "keyword",
                "ignore_above": 256
                },
            "img_url": {
                "type": "keyword",
                "ignore_above": 512 
                },
            }
        }

# --- 4. Create the Index -------------------------------------------------

INDEX_NAME = 'prod_names_for_search_hybrid'

import time
es.options(ignore_status=[400, 404]).indices.delete(index=INDEX_NAME, ignore_unavailable=True)
time.sleep(3) # Give a moment for index deletion to propagate

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, mappings=index_mapping)
    print("Index created.")
else:
    print("Index already exists.")
# -----------------------------------------------------------------------

# Indexing product data ------------------------------------------------- 
product_names = df_docs['ProductName'].apply(lambda x: x.lower()[:256]).to_list()

i = 1
for name in product_names:
    print(f"Indexing:{i} {name}")
    i += 1
    
    vector = model.encode(f"passage: {name}").tolist()
    doc = {
        "Product_name": name,
        "ProductNameVector": vector
    }
    
    es.index(index=INDEX_NAME, document=doc, refresh=True)

# ---------------------------------------------------------------------

You can check your index using diagnostic functions. Semantic search.

# --- Diagnostic functions  ---------------------------------------
def indices_list():
    return [index['index'] for index in es.cat.indices(format='json')]   

def count_records(index_name):
    return es.count(index=index_name)['count']    
# -----------------------------------------------------------------

# Get indices and record count ------------------------------------
for i in indices_list():
    print(i, count_records(i))

# Get mapings
mapping = es.indices.get_mapping(index=INDEX_NAME)
fields = mapping[INDEX_NAME]['mappings']['properties']
for field, details in fields.items():
    print(f"Model {INDEX_NAME}, fields {field}: {details['type']}")
#------------------------------------------------------------------

# --- Diagnostic functions  ---------------------------------------
def indices_list():
    return [index['index'] for index in es.cat.indices(format='json')]   

def count_records(index_name):
    return es.count(index=index_name)['count']    
# -----------------------------------------------------------------

# Get indices and record count ------------------------------------
for i in indices_list():
    print(i, count_records(i))

# Get mapings
mapping = es.indices.get_mapping(index=INDEX_NAME)
fields = mapping[INDEX_NAME]['mappings']['properties']
for field, details in fields.items():
    print(f"Model {INDEX_NAME}, fields {field}: {details['type']}")
#------------------------------------------------------------------

Semantic search

Now when you have index you can search. You can read more in my post Semantic Search with Elasticsearch

# Semantic Search --- 

query_text = "yellow texas rose"
query_vector = model.encode(f"query: {query_text}").tolist()
knn_query = {
    "field": "embedding", #"ProductNameVector",
    "query_vector": query_vector,
    "k": 10,
    "num_candidates": 50
}
response = es.search(index=INDEX_NAME, knn=knn_query, source=["product_name"])
for hit in response['hits']['hits']:
    print(f"  - Product: {hit['_source']['product_name']} (Score: {hit['_score']:.4f})")

# Semantic Search --- 

query_text = "yellow texas rose"
query_vector = model.encode(f"query: {query_text}").tolist()
knn_query = {
    "field": "embedding", #"ProductNameVector",
    "query_vector": query_vector,
    "k": 10,
    "num_candidates": 50
}
response = es.search(index=INDEX_NAME, knn=knn_query, source=["product_name"])
for hit in response['hits']['hits']:
    print(f"  - Product: {hit['_source']['product_name']} (Score: {hit['_score']:.4f})")

You can perform also lexical search or finnali hybrid search.

def search_score_script(query_h):
    response = es.search(
        index=INDEX_NAME,
        body={
            "_source": ['product_name','prodidx','img_url'],
            "query": {
                "function_score": {
                    "query": {
                        "bool": {
                            "should": [
                                {
                                    "match": {
                                        "product_name": {
                                            "query": query_h,
                                            'operator': 'and',
                                            "_name": "text_match"
                                        }
                                    }
                                },
                                {
                                    "knn": {
                                        "field": "embedding",
                                        "query_vector": model.encode(f"query: {query_text}").tolist(),
                                        "k": 30,
                                        "num_candidates": 300,
                                        "_name": "semantic_search"
                                    }
                                }
                            ]
                        }
                    },                 
                }
            },
            "size": 100
        }
    )
    return response

response = search_score_script(query_h)   
products = [
         {
             "product_name": hit["_source"]["product_name"],
             "score": hit["_score"],
             "matched_queries": hit.get("matched_queries", []),
             "prodidx": hit["_source"]["prodidx"],
         }
         for hit in response["hits"]["hits"]
     ]

print([(p["product_name"],p["score"] for p in products])
 

def search_score_script(query_h):
    response = es.search(
        index=INDEX_NAME,
        body={
            "_source": ['product_name','prodidx','img_url'],
            "query": {
                "function_score": {
                    "query": {
                        "bool": {
                            "should": [
                                {
                                    "match": {
                                        "product_name": {
                                            "query": query_h,
                                            'operator': 'and',
                                            "_name": "text_match"
                                        }
                                    }
                                },
                                {
                                    "knn": {
                                        "field": "embedding",
                                        "query_vector": model.encode(f"query: {query_text}").tolist(),
                                        "k": 30,
                                        "num_candidates": 300,
                                        "_name": "semantic_search"
                                    }
                                }
                            ]
                        }
                    },                 
                }
            },
            "size": 100
        }
    )
    return response

response = search_score_script(query_h)   
products = [
         {
             "product_name": hit["_source"]["product_name"],
             "score": hit["_score"],
             "matched_queries": hit.get("matched_queries", []),
             "prodidx": hit["_source"]["prodidx"],
         }
         for hit in response["hits"]["hits"]
     ]

print([(p["product_name"],p["score"] for p in products])
 

Sentence Transformers (e.g., MiniLM, BERT variants) are the best choice models for semantic search

• Type: Dense vector models.
• Pros:
  • Rich semantic understanding.
  • Multilingual support.
  • Fine-tuning possible for domain-specific needs.
• Popular Models:
  • msmarco-MiniLM-L-12-v3 – optimized for asymmetric search (short queries vs. long product descriptions) 3.
  • all-MiniLM-L6-v2 – fast and lightweight for general semantic tasks.
• Use Case: Ideal for large-scale product catalogs and multilingual e-commerce platforms 3 4.

3. Hybrid Search (BM25 + Semantic)

• Combine BM25 (keyword relevance) with semantic embeddings using Reciprocal Rank Fusion (RRF).
• Delivers highly relevant results by balancing literal matches and contextual meaning 1.

The post Semantic Search appeared first on Customer Experience Management.

Reduce records in reference. Instead of query 1 which is quite heavy you can use query 2 – much lighter.

-- query 1
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

-- query 1
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

-- query 2
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

-- query 2
SELECT a.ProductId, p.[index] ProdIdx, p.[Name], p.[PriceDefault],
 isNull(pp.category_top,'Cat_null') Category_top, COUNT(*) Prod_bom_count
  
    FROM [DBProductAttribute] a with (nolock)
        left join [DBProduct] p with (nolock) on p.Id=a.ProductId
        left join [DB_Products] pp with (nolock) on pp.ProductId=a.ProductId
    where a.[name]='bom'
group by a.ProductId, p.[index], p.[Name], p.[PriceDefault], isNull(pp.category_top,'Cat_null')

10 Tips How To Make SQL Lighter

1. Select Only What You Need

Instead of using SELECT *, specify the exact columns you need. This reduces the amount of data transferred from the database server to the application, lowering network traffic and memory usage.

SQL

-- Bad
SELECT * FROM employees;

-- Good
SELECT employee_id, first_name, last_name FROM employees;

2. Use Joins Efficiently

Use the most appropriate join type for your needs. For example, use INNER JOIN when you only need rows with matching values in both tables. Avoid cross joins if possible, as they produce the Cartesian product of the two tables, which can be massive.

3. Filter Data Early. 10 Tips How To Make SQL Lighter.

Use the WHERE clause to filter data as early as possible. This reduces the number of rows processed by subsequent operations like joins or aggregations.

SQL

-- Bad
SELECT first_name, last_name, salary
FROM employees
WHERE department_id IN (
    SELECT department_id FROM departments WHERE location = 'New York'
);

-- Good (filtering early)
SELECT e.first_name, e.last_name, e.salary
FROM employees AS e
JOIN departments AS d ON e.department_id = d.department_id
WHERE d.location = 'New York';

4. Use Indexes

Indexes are crucial for speeding up data retrieval. Ensure that columns used in WHERE clauses, JOIN conditions, and ORDER BY clauses are indexed. However, be mindful that too many indexes can slow down INSERT, UPDATE, and DELETE operations.

5. Avoid Subqueries in the WHERE Clause

Correlated subqueries can be very slow because they execute once for each row in the outer query. It’s often more efficient to rewrite them as JOINs.

6. Use UNION ALL Instead of UNION

UNION removes duplicate rows, which is an expensive operation. If you know there are no duplicates or you don’t need to remove them, use UNION ALL for better performance.

7. Limit Your Results

When fetching a large dataset, use LIMIT (or TOP in SQL Server) to restrict the number of rows returned. This is especially useful for pagination or when you only need a sample of the data. 10 Tips How To Make SQL Lighter.

8. Be Cautious with LIKE and Functions

Using the LIKE operator with a leading wildcard (%) or applying functions to columns in a WHERE clause can prevent the database from using an index. Instead of '%value', use 'value%'. Avoid WHERE UPPER(column) = 'VALUE', and try to store data in a consistent case.

9. Use EXISTS Instead of IN

For subqueries that check for the existence of rows, EXISTS is often faster than IN because EXISTS stops scanning as soon as it finds a match, whereas IN might scan the entire subquery result set.

10. Use Appropriate Data Types

Choosing the right data type for your columns can save storage space and improve performance. For example, using TINYINT instead of INT for a small numerical range, or CHAR instead of VARCHAR if the length is fixed.

Some tips consulted with https://gemini.google.com

More sql tips here

The post 10 Tips How To Make SQL Lighter appeared first on Customer Experience Management.

Python’s powerful date and time functions using the datetime and pandas libraries gives you a robust date table ready for Power BI and other business intelligence and analytical tools. Python for analysts most important datetime functions.

Mastering Date and Time Functions in Python for Power BI Date Tables

When working with Power BI, a well-structured Date Table is essential for time intelligence calculations like YTD, QTD, MTD, and custom period comparisons. While Power BI has built-in date table features, using Python to generate a custom date table gives you full control over the structure, granularity, and logic.

In this post, we’ll explore Python’s powerful date and time functions using the datetime and pandas libraries, and show how to create a robust date table ready for Power BI.

Python for analyst – date and time functions – basics

Python provides the datetime module to work with dates and times. Here’s a quick overview:

from datetime import datetime, timedelta, date

# Current date and time
now = datetime.now()
print("Now:", now)

# Just the date
today = date.today()
print("Today:", today)

# Add 7 days
next_week = today + timedelta(days=7)
print("Next week:", next_week)

# Subtract 30 days
last_month = today - timedelta(days=30)
print("30 days ago:", last_month)

These functions are the foundation for generating date ranges and calculating custom columns like fiscal periods or holidays. Python for analysts most important datetime functions.

Creating a Date Range with Pandas

To build a date table, we need a continuous range of dates. pandas.date_range() is perfect for this:

import pandas as pd

# Generate a date range from 2020 to 2030
date_range = pd.date_range(start='2020-01-01', end='2030-12-31', freq='D')
df = pd.DataFrame({'Date': date_range})

This gives us a DataFrame with one row per day — the backbone of our date table.

Enriching the Date Table

Now let’s add useful columns for Power BI:

df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['MonthName'] = df['Date'].dt.strftime('%B')
df['Quarter'] = df['Date'].dt.quarter
df['Day'] = df['Date'].dt.day
df['Weekday'] = df['Date'].dt.weekday + 1  # Monday = 1
df['WeekdayName'] = df['Date'].dt.strftime('%A')
df['IsWeekend'] = df['Weekday'].isin([6, 7])
df['Week'] = df['Date'].dt.isocalendar().week
df['DayOfYear'] = df['Date'].dt.dayofyear

These columns allow for slicing and dicing your data in Power BI by year, month, weekday, and more.

Fiscal Calendar Support

Many businesses use fiscal calendars that don’t align with the calendar year. Here’s how to add a fiscal year starting in July:

df['FiscalYear'] = df['Date'].apply(lambda x: x.year if x.month < 7 else x.year + 1)
df['FiscalMonth'] = df['Date'].apply(lambda x: x.month - 6 if x.month >= 7 else x.month + 6)
df['FiscalQuarter'] = ((df['FiscalMonth'] - 1) // 3) + 1

This logic adjusts the fiscal year, month, and quarter based on a July start. Python for analysts most important datetime functions

5. Flags for Time Intelligence

Power BI benefits from flags that simplify DAX calculations:

today = pd.to_datetime('today').normalize()

df['IsToday'] = df['Date'] == today
df['IsCurrentMonth'] = (df['Date'].dt.month == today.month) & (df['Date'].dt.year == today.year)
df['IsCurrentYear'] = df['Date'].dt.year == today.year

You can also add flags for holidays, fiscal periods, or custom business logic. Python for analysts most important datetime functions

Exporting to CSV for Power BI

Once your date table is ready, export it:

df.to_csv('DateTable.csv', index=False)

You can now import this CSV into Power BI as a static date table.

7. Sample Output

Here’s a preview of what your date table might look like:

Pandas to_datetime() function

Python for analysts most important datetime functions – pandas

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  object        
 3   Dt_last_purchase        13483 non-null  object        

df_cust['Dt_first_purchase'] = pd.to_datetime(df_cust['Dt_first_purchase'],format="yyyy-mm-dd")
df_cust['Dt_last_purchase'] = pd.to_datetime(df_cust['Dt_last_purchase'],format="yyyy-mm-dd")

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  datetime64[ns]
 3   Dt_last_purchase        13483 non-null  datetime64[ns]

Advanced Tips

• Holidays: Use external APIs or CSVs to mark public holidays.
• Week Start: Adjust Weekday to match your locale (e.g., Monday vs. Sunday).
• Time Zones: Use pytz or zoneinfo for timezone-aware datetime handling.
• Dynamic Updates: Automate the script to regenerate the table monthly or yearly.

Certainly! Here’s a concise 400-word post on SQL Date and Time Functions, with examples, tailored for building a Date Table in Power BI:

SQL Date and Time Functions for Power BI Date Tables

When building reports in Power BI, a comprehensive Date Table is essential for enabling time-based calculations like YTD, MTD, and custom period comparisons. While Power BI can auto-generate a date table, using SQL to create one gives you full control over its structure and logic.

Let’s explore key SQL Server date and time functions and how to use them to build a robust date table.

1. Generating a Date Range

To create a date table, you need a continuous range of dates. In SQL Server, you can use a loop or a recursive CTE:

DECLARE @StartDate DATE = '2020-01-01';
DECLARE @EndDate DATE = '2030-12-31';

WITH DateCTE AS (
    SELECT @StartDate AS DateValue
    UNION ALL
    SELECT DATEADD(DAY, 1, DateValue)
    FROM DateCTE
    WHERE DateValue < @EndDate
)
SELECT * INTO DateTable FROM DateCTE
OPTION (MAXRECURSION 32767);

select * from DateTable

Here the example

This creates a table with one row per day between 2020 and 2030.

2. Adding Date Attributes

Once you have the base dates, enrich them with useful columns:

ALTER TABLE DateTable ADD 
    Year INT,
    Month INT,
    MonthName VARCHAR(20),
    Quarter INT,
    Weekday INT,
    WeekdayName VARCHAR(20);

UPDATE DateTable
SET 
    Year = YEAR(DateValue),
    Month = MONTH(DateValue),
    MonthName = DATENAME(MONTH, DateValue),
    Quarter = DATEPART(QUARTER, DateValue),
    Weekday = DATEPART(WEEKDAY, DateValue),
    WeekdayName = DATENAME(WEEKDAY, DateValue);

These columns allow for flexible filtering and grouping in Power BI.

3. Fiscal Calendar and Flags

You can also add fiscal logic and flags:

ALTER TABLE DateTable ADD FiscalYear INT;

UPDATE DateTable
SET FiscalYear = CASE 
    WHEN MONTH(DateValue) >= 7 THEN YEAR(DateValue) + 1
    ELSE YEAR(DateValue)
END;

Add flags like IsWeekend, IsToday, or IsCurrentMonth to simplify DAX expressions.

Conclusion

Python offers a flexible and powerful way to create a custom date table for Power BI. With just a few lines of code, you can generate a rich dataset that supports advanced time intelligence and reporting needs.

Whether you’re working with fiscal calendars, custom flags, or multilingual support, Python gives you the tools to tailor your date table exactly to your business requirements.

SQL’s date and time functions like DATEADD, DATEPART, DATENAME, and YEAR are powerful tools for building a custom date table. Once created, export it to Power BI or use it as a view for dynamic reporting.

Would you like a ready-to-run SQL script for a complete date table?

Find more resources in our course of Advanced programming for business analysts

The post Python for analysts most important datetime functions appeared first on Customer Experience Management.

Post is based on: Calculate a Moving Average with T-SQL Windowing Functions

-- test table
-- mssqltips.com
DROP TABLE IF EXISTS dbo.BigWeightTracker;
GO

CREATE TABLE dbo.BigWeightTracker
(
    Id INT IDENTITY(1, 1) NOT NULL,
    UserId INT NOT NULL,
    Pounds Decimal(10, 2) NOT NULL,
    DateRecorded DATE NOT NULL,
    CONSTRAINT PK_BigWeightTracker_Id
        PRIMARY KEY CLUSTERED (Id)
);
GO

DECLARE @StartDate DATE = '2022-01-01';
DECLARE @EndDate DATE = '2023-12-31';

WITH 
 Users AS (SELECT TOP 60
        ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS UserId
    -- select top 100 *
    FROM sys.all_columns AS s1
   ),
 Dates AS (
    SELECT @StartDate AS Date
    UNION ALL
    SELECT DATEADD(DAY, 1, Date) AS Date FROM Dates  WHERE Date < @EndDate
   )
--select * from Dates, ale insert działa bez tego ograniczenia

INSERT INTO dbo.BigWeightTracker
(
    UserId,
    DateRecorded,
    Pounds
)
SELECT u.UserId,
       d.Date,
       (ABS(CHECKSUM(NEWID()) % (220 - 170 + 1)) + 170) AS Weight
FROM Users u
    CROSS JOIN Dates d
OPTION (MAXRECURSION 0);
GO

-- koniec tworzenia tabeli -----------

Indexing

-- indexing
DROP INDEX IF EXISTS IX_BigWeightTracker_Average ON dbo.BigWeightTracker;

CREATE NONCLUSTERED INDEX IX_BigWeightTracker_Average
ON dbo.BigWeightTracker
(
    UserId,
    DateRecorded ASC
)
INCLUDE (Pounds);
GO

moving average – old approach

-- moving avg - old aproach
-- mssqltips.com
SET STATISTICS TIME, IO ON;
SELECT t1.UserId,
       t1.DateRecorded,
       t1.Pounds,
       CASE
           WHEN COUNT(t2.Pounds) < 7 THEN
               NULL
           ELSE
               AVG(t2.Pounds)
       END AS [7DayMovingAverage]
FROM dbo.BigWeightTracker t1
    INNER JOIN dbo.BigWeightTracker t2
        ON t2.DateRecorded
           BETWEEN DATEADD(DAY, -6, t1.DateRecorded) AND t1.DateRecorded
           AND t1.UserId = t2.UserId
GROUP BY t1.UserId,
         t1.DateRecorded,
         t1.Pounds
ORDER BY t1.UserId,
         t1.DateRecorded;
SET STATISTICS TIME, IO OFF;
GO

Statistics

(438000 rows affected)
Tabela „BigWeightTracker”. 
Liczba skanowań: 10, odczyty logiczne: 3310, odczyty z wyprzedzeniem: 1542.

Tabela „Worktable”. Liczba skanowań: 600, odczyty logiczne: 2634596, odczyty z wyprzedzeniem: 10957.
SQL Server — czasy wykonywania:
  Czas procesora CPU = 94376 ms; upłynęło czasu = 27683 ms.

Another way to write this query is to use a correlated subquery (podzapytanie). Based on the performance metrics, this approach still results in a high number of logical reads and scans with about the same execution time. Moving average in SQL.

-- qwerenda z podzapytaniem
-- mssqltips.com
SET STATISTICS TIME, IO ON;
SELECT t1.UserId,
       t1.DateRecorded,
       t1.Pounds,
       (
           SELECT CASE
                      WHEN COUNT(t2.Pounds) < 7 THEN
                          NULL
                      ELSE
                          AVG(t2.Pounds)
                  END
           FROM BigWeightTracker t2
           WHERE t2.DateRecorded
                 BETWEEN DATEADD(DAY, -6, t1.DateRecorded) AND t1.DateRecorded
                 AND t1.UserId = t2.UserId
       ) AS [7DayMovingAverage]
FROM dbo.BigWeightTracker t1
ORDER BY t1.UserId,
         t1.DateRecorded;
SET STATISTICS TIME, IO OFF;

Statystyka

(438000 rows affected)
Tabela „BigWeightTracker”. Liczba skanowań: 6, odczyty logiczne: 3232, odczyty fizyczne: 3, odczyty z wyprzedzeniem: 1573.
Tabela „Worktable”. Liczba skanowań: 1314000, odczyty logiczne: 6884158.
 SQL Server — czasy wykonywania:
  Czas procesora CPU = 8593 ms; upłynęło czasu = 5162 ms.

Window functions

Microsoft introduced window functions in SQL Server 2005, but added new and improved options in SQL Server 2012. A few functions to use with windowing include:

• ROW_NUMBER: Adds a sequential number to each row.
• RANK: Ranks rows and leaves a gap when there are ties (rows values equal).
• DENSE_RANK: Like RANK but doesn’t leave gaps when there are ties.

The OVER clause allows you to define your window frame. Additionally, you can tell the frame how many prior rows to include with the ROWS PRECEDING argument. Before we build the 7-day average, let’s look at a simple example. Moving average in SQL.

-- simple test
-- mssqltips.com
DECLARE @Simple_Test AS TABLE (Id INT);
INSERT INTO @Simple_Test
(Id) VALUES (1),(2),(3),(4),(5);

SELECT Id,
       SUM(Id) OVER (ORDER BY Id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS Value
FROM @Simple_Test;
GO

To achieve moving average, use the argument ROWS BETWEEN 1 PRECEDING AND CURRENT ROW.

-- mssqltips.com
SET STATISTICS TIME, IO ON;
SELECT t1.UserId,
       t1.DateRecorded,
       t1.Pounds,
       AVG(t1.Pounds) 
         OVER (
           PARTITION BY t1.UserId 
              ORDER BY t1.DateRecorded 
                 ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS [7DayMovingAverage]
FROM dbo.BigWeightTracker t1
ORDER BY t1.UserId,
         t1.DateRecorded;
SET STATISTICS TIME, IO OFF;

Statistics. Moving average in SQL

(438000 rows affected)
Tabela „BigWeightTracker”. Liczba skanowań: 5, odczyty logiczne: 1655,

 SQL Server — czasy wykonywania:
  Czas procesora CPU = 3079 ms; upłynęło czasu = 2382 ms.

Key Points of Moving average in SQL

• Windowing functions are an excellent solution for use cases like moving average, where you might otherwise perform self-joins or correlated subqueries to pull back the results.
• Additional arguments use in windowing function ROWS BETWEEN n PRECEDING AND CURRENT ROW.

Anomaly Detection for User Sessions and Logins using Moving average in SQL

-- ===================================================================================
-- Anomaly Detection for User Sessions and Logins
--
-- Purpose: This query identifies days with unusual session or login counts.
-- Method:  It flags a day as an anomaly if its count is more than 2 standard
--          deviations away from the 5-day moving average.
-- ===================================================================================

WITH RunningStats AS (
    -- This CTE calculates a 5-day moving average and standard deviation for session
    -- and logged-in user counts. The window includes the current day and the four previous days.
    SELECT
        [Probe_date],
        [User_internet],
        [User_logged],
        [Sesion_count],
        AVG(CAST(Sesion_count AS FLOAT)) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_avg_session,
        STDEV(Sesion_count) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_std_session,
        AVG(CAST(User_logged AS FLOAT)) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_avg_logged,
        STDEV(User_logged) OVER (ORDER BY [Probe_date] ROWS BETWEEN 4 PRECEDING AND CURRENT ROW) AS moving_std_logged
    FROM
        [DBSession_stat]
    -- Restrict the analysis to the last 30 days for relevance.
    WHERE
        [Probe_date] >= DATEADD(day, -30, GETUTCDATE())
)
-- Final step: Select all calculated stats but only for the days flagged as anomalous.
SELECT
    *
FROM
    RunningStats
-- The WHERE clause applies the "2-sigma" rule to filter for anomalies. A day is
-- returned if either the session count or the logged-in user count is anomalous.
WHERE
    -- Condition for session anomaly
    ABS(Sesion_count - moving_avg_session) > 2 * moving_std_session
    OR
    -- Condition for logged-in user anomaly
    ABS(User_logged - moving_avg_logged) > 2 * moving_std_logged;

Here another approach comparing also weekly pattern.

import pandas as pd

# --- 1. Load and Prepare the Data ---

# Load the data from the uploaded CSV file
df = pd.read_csv('Sesion_stat_last_30_days_an_sessions.xlsx - Arkusz4.csv')

# Convert 'Probe_date' to datetime objects for time-based analysis
df['Probe_date'] = pd.to_datetime(df['Probe_date'])

# Set the 'Probe_date' as the index of the DataFrame
df = df.set_index('Probe_date').sort_index()


# --- 2. Feature Engineering: Create Baselines for Comparison ---

# Define the number of data points in a week (7 days * 24 hours * 6 10-minute intervals)
# This is used to look back at the same time last week.
PREVIOUS_WEEK_SHIFT = 7 * 24 * 6

# Calculate the 1-hour running average (6 data points * 10 minutes = 60 minutes)
df['running_avg'] = df['Sesion_count'].rolling(window=6, min_periods=1).mean()

# Get the session count from the exact same time one week prior
df['prev_week_sessions'] = df['Sesion_count'].shift(PREVIOUS_WEEK_SHIFT)


# --- 3. Anomaly Detection Logic ---

# Define the thresholds for what constitutes an anomaly
# A 50% increase over both the running average and the previous week's value
RELATIVE_THRESHOLD = 1.5
# The session count must also be at least this high to be considered
ABSOLUTE_THRESHOLD = 50

# Apply the anomaly detection conditions
df['is_anomaly'] = (
    (df['Sesion_count'] > (df['running_avg'] * RELATIVE_THRESHOLD)) &
    (df['Sesion_count'] > (df['prev_week_sessions'] * RELATIVE_THRESHOLD)) &
    (df['Sesion_count'] > ABSOLUTE_THRESHOLD)
)


# --- 4. Review the Results ---

# Select only the rows where an anomaly was detected
anomalies = df[df['is_anomaly']].copy()

# Add columns to show the deviation that triggered the flag
anomalies['pct_over_avg'] = (anomalies['Sesion_count'] / anomalies['running_avg'] - 1) * 100
anomalies['pct_over_prev_week'] = (anomalies['Sesion_count'] / anomalies['prev_week_sessions'] - 1) * 100


print("Anomalies Detected:")
# Display the identified anomalies with context
print(anomalies[[
    'Sesion_count',
    'running_avg',
    'prev_week_sessions',
    'pct_over_avg',
    'pct_over_prev_week'
]].round(1))

Program detected following anomalies

Moving average in SQL

Read more Python for analysts most important datetime functions

The post Moving average in SQL appeared first on Customer Experience Management.

Course Structure:

Week 1-2: Introduction and Setup

• Lecture: Overview of SQL and Python in business analytics.
• Lab: Setting up the environment (SQL databases, Python IDEs).
• Project 1: Data Import and Cleaning
  • SQL: Importing data from various sources, cleaning and preprocessing.
  • Python: Using pandas for data cleaning and manipulation.

Week 3-4: Data Exploration and Visualization

• Lecture: Techniques for data exploration and visualization.
• Lab: SQL queries for data exploration, Python libraries for visualization (matplotlib, seaborn).
• Project 2: Exploratory Data Analysis (EDA)
  • SQL: Writing complex queries to explore data.
  • Python: Visualizing data trends and patterns.

Week 5-6: Statistical Analysis

• Lecture: Statistical methods for business analytics.
• Lab: SQL functions for statistical analysis, Python libraries (numpy, scipy).
• Project 3: Statistical Analysis of Sales Data
  • SQL: Calculating statistical measures (mean, median, standard deviation).
  • Python: Performing hypothesis testing and regression analysis.

Week 7-8: Predictive Modeling

• Lecture: Introduction to predictive modeling techniques.
• Lab: SQL for data preparation, Python for model building (scikit-learn).
• Project 4: Predictive Sales Forecasting
  • SQL: Preparing data for modeling.
  • Python: Building and evaluating predictive models.

Week 9-10: Time Series Analysis

• Lecture: Time series analysis and forecasting.
• Lab: SQL for time series data manipulation, Python libraries (statsmodels).
• Project 5: Time Series Forecasting
  • SQL: Extracting and transforming time series data.
  • Python: Building time series models and forecasting.

Week 11-12: Machine Learning Integration

• Lecture: Integrating machine learning with SQL databases.
• Lab: SQL for data storage and retrieval, Python for machine learning (TensorFlow, Keras).
• Project 6: Customer Segmentation
  • SQL: Storing and retrieving data for machine learning.
  • Python: Building and deploying machine learning models.

Week 13-14: Advanced Topics and Final Project

• Lecture: Advanced topics in SQL and Python (optimization, big data).
• Lab: SQL performance tuning, Python for big data (PySpark).
• Project 7: Comprehensive Business Analytics Project
  • SQL: Optimizing queries for large datasets.
  • Python: Analyzing and visualizing large datasets.

Assessment:

• Projects: Each mini project will be assessed based on accuracy, efficiency, and creativity.
• Final Project: A comprehensive project integrating all learned skills.

Resources:

• Books: “SQL for Data Analytics” by Upom Malik, Matt Goldwasser, and Benjamin Johnston; “Python for Data Analysis” by Wes McKinney.
• Online Tutorials: SQLZoo, DataCamp, Coursera.

Advanced Programming in SQL and Python

Advanced Programming in Data Analysis

https://datascience.umcs.pl

The post Advanced Programming in SQL and Python – the course for business analysts appeared first on Customer Experience Management.