The “Marketplace Deduplication” Problem

Context: Large marketplaces (Amazon, eBay, Alibaba, or a niche aggregator) allow thousands of third-party sellers to upload their own product feeds. Product Grouping for E-commerce Datasets indeed.

The Problem:

• Seller A uploads: “Apple iPhone 13, 128GB, Midnight”
• Seller B uploads: “iPhone 13 128 GB Black Unlocked”
• Seller C uploads: “Smartfon Apple iPhone 13 128GB (MLPF3PM/A)”

The “Omnichannel Customer Stitching” Problem (Single Customer View)

Context: A retailer sells through a Website, a Mobile App, and Physical Stores. They want to know if the person browsing the app is the same person buying in the store.

The Problem:

• Record A (Online): email: j.smith@gmail.com, cookie_id: xyz123, behavior: viewed running shoes
• Record B (In-Store POS): card_hash: ****-1234, loyalty_id: 998877, name: John Smith
• Record C (Customer Support): phone: +48 500..., name: Johnny Smith, complaint: "Shoes size 42 too small"

Why it’s hard:

The “Competitor Price Monitoring” Problem

Context: An e-commerce store wants to automatically adjust their prices to be $1 cheaper than their biggest competitor.

The Problem:

• Your Product: “Samsung Galaxy S20 FE 5G Cloud Navy”
• Competitor Site: “Samsung S20 Fan Edition (Navy) – 5G Compatible”

Why it’s hard:

Abstract

• Problem: E-commerce catalogs suffer from redundancy (same product, different sizes/variants).
• Gap: Manual grouping is impossible; Deep Learning is overkill/imprecise for strict SKU grouping.
• Method: We compare Jaccard (Set) vs. TF-IDF (Vector) and propose a normalization pipeline.
• Result: Our method achieved X% accuracy with Y% reduction in computational time.

Set-Theoretic Approaches for Product Grouping (Jaccard)

• Concept: Treats text as a “Bag of Words” (BoW) without weights.
• Key Papers/Concepts:
  • Cohen et al. (2003) often discuss string metrics for entity matching.
  • Shingling / MinHash: In large datasets, calculating Jaccard for all pairs is $O(N^2)$. Literature focuses on Locality Sensitive Hashing (LSH) (MinHash) to approximate Jaccard similarity efficiently.
  • Pros in Literature: High interpretability, excellent for “near-duplicate” detection.
  • Cons: Fails when synonyms are used (e.g., “pants” vs “trousers”) or when word importance varies.

Vector Space Models (TF-IDF)

• Concept: Maps text to a high-dimensional Euclidean space.
• Key Papers/Concepts:
  • Salton et al. (1975) (The foundational VSM paper).
  • Character n-grams: Papers often cite that for noisy user-generated content (UGC), character n-grams outperform word tokens because they handle misspellings morphologically.
  • Pros: Handles “rare words” (like model numbers) better due to IDF (Inverse Document Frequency) weighting.

The State-of-the-Art (Deep Learning / Embeddings)

• If you publish, reviewers will ask: “Why not BERT?”
• SBERT (Sentence-BERT): Current SOTA uses transformer models to generate dense vector embeddings.
• Your Counter-Argument: Deep learning is computationally expensive and “black-box”. For industrial product grouping where exact feature matching (like “Samsung” + “Galaxy”) is critical, classical methods (TF-IDF/Jaccard) often offer better precision and control than semantic embeddings which might group “iPhone 12” with “Samsung S20” because they are both “phones”.

II. Related Work

• Mention LSH (Locality Sensitive Hashing) for Jaccard.
• Mention DBSCAN and Agglomerative Clustering as standard algorithms.
• Cite limitations of BERT in high-precision SKU matching.
• Product Grouping for E-commerce

III. Methodology (The Core)

1. Preprocessing – The Normalization Filter for Product Grouping for E-commerce:
   • Define your Regex rules mathematically.
   • $T_{clean} = f(T_{raw})$ where $f$ removes tokens $t \in \{Dimensions, Colors, Stopwords\}$.
2. Representation:
   • Approach A (Jaccard): $J(A,B) = \frac{|A \cap B|}{|A \cup B|}$
   • Approach B (TF-IDF): Cosine Similarity $\cos(\theta) = \frac{A \cdot B}{||A|| ||B||}$
3. Clustering Algorithm:
   • Explain why you chose Connected Components (Graph theory) for Jaccard or Hierarchical Clustering for TF-IDF.

IV. Experiments (Product Grouping for E-commerce)

• Dataset: Your dataset of “several thousand products”.
• Metrics: You must measure quality.
  • Precision: Are elements in the cluster actually the same product?
  • Recall: Did we find all sizes of that product?
  • F1-Score: Harmonic mean of the two.

V. Results & Discussion

• Hypothesis: Jaccard works better for clean data; TF-IDF works better for noisy data (typos).
• Observation: Jaccard is faster but TF-IDF + N-grams is more robust.

Novel Algorithm:

1. Stage 1 (Blocking): Use Jaccard on tokens to quickly find “Candidate Pairs” (fast, rough filter).
2. Stage 2 (Refinement): Use TF-IDF with Character N-grams on the candidates to calculate a precise similarity score (handles typos).
3. Stage 3 (Decision): Hard threshold (e.g., >0.85).

Relevant Search Terms & Papers

Entity Resolution (ER) is the problem of identifying which records in a database refer to the same real-world entity. An exhaustive ER process involves computing the similarities between pairs of records, which can be very expensive for large datasets. Various blocking techniques can be used to enhance the performance of ER by dividing the records into blocks in multiple ways and only comparing records within the same block.

However, most blocking techniques process blocks separately and do not exploit the results of other blocks. In this paper (https://dl.acm.org/doi/pdf/10.1145/1559845.1559870), authors propose an iterative blocking framework where the ER results of blocks are reflected to subsequently processed blocks. Blocks are now iteratively processed until no block contains any more matching records.

Compared to simple blocking, iterative blocking may achieve higher accuracy because reflecting the ER results of blocks to other blocks may generate additional record matches. Iterative blocking may also be more efficient because processing a block now saves the processing time for other blocks.

• “Short Text Clustering for E-commerce”
• “Product Entity Resolution with Noise”
• “Comparison of Jaccard and Cosine Similarity in Text Mining”
• “Blocking techniques for Entity Resolution” (This is crucial for scaling to thousands/millions of products). Product Grouping for E-commerce.

Specific types of papers to look for:

1. Koporec et al.: Papers on combining Jaccard with other metrics. https://www.sciencedirect.com/science/article/pii/S1364815225002981
2. Ganesan et al.: Research on “abstractive summarization” or short-text clustering in retail.

FAQ: Product Grouping in Noisy E-Commerce Datasets

The post Product Grouping in Noisy E-commerce Datasets appeared first on Customer Experience Management.

SQL is the backbone of data-driven decision-making, and mastering it can dramatically improve how you interact with databases. Whether you’re a seasoned developer or just starting out, writing efficient, readable, and scalable SQL queries is a skill that pays off daily. In this post, I’ve compiled ten essential tips that will help you write smarter SQL—tips that I’ve learned, refined, and shared over time. These aren’t just theoretical best practices; they’re practical techniques that can make your queries faster, your code cleaner, and your debugging easier.

You can try Microsoft SQL Server from here: https://www.microsoft.com/pl-pl/sql-server/sql-server-downloads. And developer edition is here

From avoiding SELECT * to choosing the right join types, each tip is designed to help you think critically about how your queries perform and how they scale. You’ll also learn how to use indexes effectively, filter data early, and make smart choices between EXISTS and IN. Each section includes a short summary and a link to a full post where you can dive deeper into the topic. Whether you’re optimizing a legacy system or building something new, these tips will help you get the most out of SQL—and avoid common pitfalls that slow down your work.

Select Only What You Need, SQL Tips no 1

Avoid SELECT * and specify only the columns you need. This reduces data transfer, memory usage, and improves query speed. For example, instead of pulling all employee data, just select employee_id, first_name, and last_name. Read more

Optimize Join Conditions for Performance

Avoid non-SARGable joins that prevent index usage. Instead of applying functions to columns in join conditions, restructure the logic to preserve index efficiency. This dramatically improves query speed. Read the full guide

Use the PIVOT Operator for Better Reporting

Transform row-based data into columnar format using PIVOT. This is ideal for cross-tab reports and trend analysis, especially when comparing metrics across time or categories. Explore the PIVOT tutorial

Master Recursive CTEs for Hierarchical Data

Recursive Common Table Expressions (CTEs) allow you to elegantly query hierarchical or tree-structured data. They’re powerful for tasks like organizational charts or category trees. Learn about recursive CTEs

Set the First Day of the Week with DATEFIRST

Use SET DATEFIRST to control how SQL Server interprets weekday numbers. This is crucial for accurate time-based reporting and week-based aggregations. See how to use DATEFIRST

Update Multiple Tables with Conditions

Learn how to structure multi-table updates using joins and conditional logic. This technique is essential for synchronizing data across related tables. Read the multi-table update example

Filter Early with WHERE Clauses

Apply filters as early as possible to reduce the number of rows processed in joins and aggregations. Optimize your filtering

Use UNION ALL Instead of UNION

UNION ALL is faster than UNION because it skips duplicate elimination. Use it when duplicates aren’t a concern. Performance tip explained

Avoid Functions on Indexed Columns

Using functions like LOWER() or DATEADD() on indexed columns disables index usage. Rewrite conditions to preserve index paths. Join optimization example

Explore SQL for Business Analytics

Advanced SQL techniques like statistical analysis, predictive modeling, and time series forecasting are essential for business analysts. Learn how to combine SQL with Python for deeper insights. Check out the full course

5 additional SQL tips

Use CTEs for Readability and Reuse

Common Table Expressions (CTEs) make complex queries easier to read and maintain. They allow you to define temporary result sets that can be referenced multiple times. SQL Tips

WITH recentorders AS (
  SELECT customerid, orderdate
  FROM orders
  WHERE orderdate > CURRENTDATE - INTERVAL '30 days'
)
SELECT customerid, COUNT(*) AS ordercount
FROM recentorders
GROUP BY customer_id;

Avoid Functions on Indexed Columns

Using functions on indexed columns disables index usage, slowing down queries. Instead, transform the value before comparison. SQL Tips

-- Avoid
SELECT FROM users WHERE LOWER(email) = 'test@example.com';
-- Better
SELECT FROM users WHERE email = 'test@example.com';

Use CASE for Conditional Logic

CASE lets you embed conditional logic directly in your queries, useful for categorizing or transforming data.

SELECT name,
  CASE
    WHEN score >= 90 THEN 'Excellent'
    WHEN score >= 75 THEN 'Good'
    ELSE 'Needs Improvement'
  END AS performance
FROM students;

Optimize Aggregations with GROUP BY

When aggregating data, ensure you’re grouping only necessary columns to avoid performance hits and incorrect results.

SELECT department, AVG(salary) AS avg_salary
FROM employees
GROUP BY department;

Use Parameterized Queries to Prevent SQL Injection

Always use parameterized queries in application code to protect against SQL injection.

-- Example in Python with psycopg2
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))

Remember – tips summary

Use Joins Efficiently

Choose the right join type—INNER JOIN for matched rows, and avoid CROSS JOIN unless necessary. Efficient joins reduce unnecessary data processing and improve clarity.

Filter Data Early

Apply filters as soon as possible in your query using WHERE clauses. This minimizes the number of rows processed in joins and aggregations, leading to faster execution. SQL Tips no 3

Use Indexes Wisely

Indexes speed up data retrieval, especially in WHERE, JOIN, and ORDER BY clauses. But don’t over-index—too many can slow down INSERT, UPDATE, and DELETE operations.

Avoid Subqueries in WHERE Clauses

Correlated subqueries can be slow. Replace them with joins when possible to improve performance and readability. SQL Tips no 4

Use UNION ALL Instead of UNION

UNION removes duplicates, which is costly. If duplicates aren’t a concern, use UNION ALL for faster results.

Limit Your Results

Use LIMIT or TOP to restrict the number of rows returned. This is especially useful for pagination or sampling large datasets.

Be Cautious with LIKE and Functions

Avoid leading wildcards in LIKE and functions in WHERE clauses—they prevent index usage. Instead, use indexed columns and consistent casing.

Use EXISTS Instead of IN

EXISTS is often faster than IN because it stops scanning once a match is found. Use it for subqueries checking row existence.

Use Appropriate Data Types

Choosing the right data type—like TINYINT over INT or CHAR over VARCHAR—can save space and improve performance.

SQL Server Management Studio

SSMS as a Comprehensive SQL Environment
SQL Server Management Studio (https://learn.microsoft.com/en-us/ssms/) is a powerful, integrated environment for managing SQL Server infrastructure. It provides tools for writing, executing, and debugging SQL queries, as well as managing databases, tables, views, and stored procedures. SSMS supports both on-premises and cloud-based SQL Server instances, making it versatile for hybrid environments. Its intuitive interface includes Object Explorer for navigating server components and Query Editor for crafting and testing SQL scripts. Whether you’re a database administrator or developer, SSMS offers a unified workspace that streamlines daily tasks and enhances productivity through built-in templates, syntax highlighting, and error diagnostics.

Security, Performance, and Monitoring Tools
SSMS includes robust features for security management, such as configuring roles, permissions, and auditing access. It also provides performance tuning tools like the Database Engine Tuning Advisor and graphical execution plans to help identify bottlenecks. With Activity Monitor, users can track real-time server performance, view active sessions, and analyze resource usage. These tools empower teams to maintain optimal database health and ensure compliance with organizational policies. SSMS also integrates with SQL Server Agent for scheduling jobs and alerts, making it a central hub for automation and proactive monitoring across enterprise environments.

Integration, Extensibility, and Collaboration
SSMS supports integration with source control systems like Git, enabling versioning and collaborative development. It allows exporting and importing data via wizards, scripting database objects, and generating reports for documentation. Users can extend SSMS functionality through add-ins and connect to Azure services for cloud-based analytics and storage. Its support for multiple query windows and tabbed editing enhances multitasking, while customizable keyboard shortcuts and themes improve user experience. SSMS continues to evolve with regular updates, ensuring compatibility with the latest SQL Server features and providing a stable platform for modern data operations.

The post 15 Essential SQL Tips You Can’t Live Without appeared first on Customer Experience Management.

The query to be optimized

Here there is a query. The table has no index. How can we optimize the query and the table reading performance setting up indices? select k.*, 

select k.*, 
       c.[Top_segment]
      ,c.[CompanyType]
      ,c.[Potencial] from (
    select 
          a.[Dt_ym]
          ,a.[CustomerIdEx]
          ,sum(a.ErpRegisterDoc_count) ErpRegisterDoc_count
          ,sum(a.ErpValueNet_sum) ErpValueNet_sum, sum(b.ErpValueNet_sum) ErpValueNet_sum_py
    FROM [onn].[DBCust_stat_baskets] a with (nolock)
      left join [onn].[DBCust_stat_baskets] b with (nolock) 
          on dateadd(year, 1, b.dt_ym) = a.Dt_ym and b.[CustomerIdEx] = a.CustomerIdEx
group by           a.[Dt_ym]
          ,a.[CustomerIdEx]
 ) as k 
  left join [onn].[DBCust_erp] c with (nolock) on k.CustomerIdEx = c.Cust_nr

select k.*, 
       c.[Top_segment]
      ,c.[CompanyType]
      ,c.[Potencial] from (
    select 
          a.[Dt_ym]
          ,a.[CustomerIdEx]
          ,sum(a.ErpRegisterDoc_count) ErpRegisterDoc_count
          ,sum(a.ErpValueNet_sum) ErpValueNet_sum, sum(b.ErpValueNet_sum) ErpValueNet_sum_py
    FROM [onn].[DBCust_stat_baskets] a with (nolock)
      left join [onn].[DBCust_stat_baskets] b with (nolock) 
          on dateadd(year, 1, b.dt_ym) = a.Dt_ym and b.[CustomerIdEx] = a.CustomerIdEx
group by           a.[Dt_ym]
          ,a.[CustomerIdEx]
 ) as k 
  left join [onn].[DBCust_erp] c with (nolock) on k.CustomerIdEx = c.Cust_nr

The biggest performance issues here are the complete lack of indexes and a join condition that prevents the use of an index (even if one existed). You can dramatically speed this up by creating the right indexes and making a small but critical change to the query.

Setting Up the Right Indices

An index acts like a book’s index, allowing SQL Server to find data quickly without scanning the entire table. Based on your query’s JOIN and GROUP BY clauses, these are the indexes you need.

On the DBCust_stat_baskets table:

This table is joined to itself and grouped by CustomerIdEx and Dt_ym. A covering index will be most effective here. This special type of index not only includes the key columns for searching but also the data columns the query needs, allowing the entire subquery to be satisfied from the index alone.

SQL

CREATE NONCLUSTERED INDEX IX_DBCust_stat_baskets_Customer_Date
ON [onn].[DBCust_stat_baskets] ([CustomerIdEx], [Dt_ym])
INCLUDE ([ErpRegisterDoc_count], [ErpValueNet_sum]);

• ON ([CustomerIdEx], [Dt_ym]): This covers your JOIN and GROUP BY columns. The order is important; CustomerIdEx is more selective, so it comes first.
• INCLUDE (...): This “covers” the SUM calculations, so SQL Server doesn’t have to look up the data in the main table, which is a massive performance boost.

On the DBCust_erp table:

This table is joined on the Cust_nr column. It also needs an index to make the final LEFT JOIN fast.

SQL

CREATE NONCLUSTERED INDEX IX_DBCust_erp_Cust_nr
ON [onn].[DBCust_erp] ([Cust_nr])
INCLUDE ([Top_segment], [CompanyType], [Potencial]);

• ON ([Cust_nr]): Speeds up the join to the subquery k.
• INCLUDE (...): Covers the columns you are selecting, again avoiding a second lookup to the main table.

Optimizing the T-SQL Query 🚀

Your query has a “non-SARGable” join condition. SARGable means Search Argumentable, allowing SQL to use an index.

The problem is this line: dateadd(year, 1, b.dt_ym) = a.Dt_ym. By applying the dateadd function to the column b.dt_ym, you force SQL Server to calculate a new value for every single row in the table before it can perform the join. This guarantees a slow table scan.

The fix is to apply the function to the other side of the equation, so the column itself remains “clean.”

Change this: ON dateadd(year, 1, b.dt_ym) = a.Dt_ym

To this: ON b.dt_ym = dateadd(year, -1, a.Dt_ym)

This small change allows SQL Server to take a value from a, calculate the prior year’s date once, and then perform a highly efficient index seek on b.dt_ym to find the matching row.

The Final Optimized Query

For better readability, I’ve rewritten your query using a Common Table Expression (CTE). The logic is the same, but it’s easier to follow. This version incorporates the SARGable join fix.

SQL

-- Use a CTE for better readability to get the current and prior year sums
WITH CustomerMonthlyStats AS (
    SELECT
        a.[Dt_ym],
        a.[CustomerIdEx],
        SUM(a.ErpRegisterDoc_count) AS ErpRegisterDoc_count,
        SUM(a.ErpValueNet_sum) AS ErpValueNet_sum,
        SUM(b.ErpValueNet_sum) AS ErpValueNet_sum_py -- Sum from the previous year
    FROM
        [onn].[DBCust_stat_baskets] AS a
    LEFT JOIN
       [onn].[DBCust_stat_baskets] AS b
        -- OPTIMIZED: The function is now on the outer side of the join, making it SARGable
        ON a.[CustomerIdEx] = b.[CustomerIdEx]
        AND b.dt_ym = DATEADD(year, -1, a.Dt_ym)
    GROUP BY
        a.[Dt_ym],
        a.[CustomerIdEx]
)
-- Final SELECT joining the aggregated stats with customer details
SELECT
    k.*,
    c.[Top_segment],
    c.[CompanyType],
    c.[Potencial]
FROM
    CustomerMonthlyStats AS k
LEFT JOIN
    [onn].[DBCust_erp] AS c ON k.CustomerIdEx = c.Cust_nr;

By creating these indexes and applying this simple query fix, you should see a dramatic improvement in performance, going from potentially minutes to seconds. I’d also recommend removing the WITH (NOLOCK) hints after these changes, as they won’t be necessary and can lead to reading “dirty” or incorrect data.

You also converted my query to CTE, but I cant use cte in Power BI direct query, that is why I used like that. Do I don’t know something or CTE is really better then my version of “cascade” querying?

That’s an excellent point about Power BI DirectQuery’s limitations with CTEs. You are correct; you often have to use subqueries (also called derived tables) instead.

To answer your main question: for this specific query, a CTE is not better than your subquery in terms of performance. The SQL Server query optimizer is smart enough to see that both your subquery version and the CTE version are logically identical. It will generate the exact same execution plan for both, meaning they will run at the same speed.

The primary advantage of a CTE is readability and maintainability, not performance. It helps break down complex logic into clean, sequential steps, making the query easier for a human to understand and debug later.

The Important Part: The Optimization Itself

The crucial takeaway from the previous answer was not the switch to a CTE, but these two points:

1. Creating the right indexes on the tables.
2. Fixing the non-SARGable join condition.

As long as you apply those two changes, your original query structure is perfectly fine and will be just as fast as the CTE version.

Your Original Query, Optimized

Here is your original query with the single, critical change to the JOIN condition. This version avoids CTEs and is ready for Power BI DirectQuery, while still containing the essential performance fix.

SQL

SELECT
    k.*,
    c.[Top_segment],
    c.[CompanyType],
    c.[Potencial]
FROM (
    -- This subquery is functionally the same as the CTE
    SELECT
        a.[Dt_ym],
        a.[CustomerIdEx],
        SUM(a.ErpRegisterDoc_count) AS ErpRegisterDoc_count,
        SUM(a.ErpValueNet_sum) AS ErpValueNet_sum,
        SUM(b.ErpValueNet_sum) AS ErpValueNet_sum_py
    FROM
        [onn].[DBCust_stat_baskets] AS a WITH (NOLOCK)
    LEFT JOIN
        [onn].[DBCust_stat_baskets] AS b WITH (NOLOCK)
        -- The single most important optimization is right here:
        ON a.CustomerIdEx = b.CustomerIdEx AND b.dt_ym = DATEADD(year, -1, a.Dt_ym)
    GROUP BY
        a.[Dt_ym],
        a.[CustomerIdEx]
) AS k
LEFT JOIN
   [onn].[DBCust_erp] AS c WITH (NOLOCK) ON k.CustomerIdEx = c.Cust_nr;

The post Query optimization with join condition appeared first on Customer Experience Management.

Frequency Analysis for Profiling

This is the most direct approach to identify the most and least selected lectures by your L6 students.

1. Filter Data: Isolate the rows (students) who selected L6.
2. Calculate Frequencies: For this subset of L6 students, count how many of them selected each of the other available lectures (L1, L2, L3, etc.).
   • Most Selected: The lectures with the highest counts.
   • Least Selected: The lectures with the lowest counts (or those not selected at all).

Association Rule Mining – Co-occurrence

This sophisticated approach, often used in market basket analysis, can determine which lectures are most frequently chosen together with L6.

• Support: The proportion of L6 students who also selected a specific lecture (e.g., L3).
• Confidence: The likelihood that a student selected L6 given that they selected another lecture (e.g., L3 → L6), or vice-versa.
• Lift: A measure of how much more likely a student is to select L3 if they also selected L6, compared to the overall likelihood of selecting L3. A Lift >1 suggests a positive association (students who take one tend to take the other).

Correlation Analysis – the Strength and Direction of Relation – application in Profiling

This method quantifies the relationship between selecting L6 and selecting any other lecture (Lx). Since the selection data is binary (0 for not selected, 1 for selected), you would use a correlation measure suitable for binary variables. Profiling.

• Phi Coefficient (ϕ): This is a measure of association for two binary variables. It ranges from −1 to +1.
  • Strong Positive Correlation (ϕ≈+1): Students who select L6 are highly likely to also select Lx. This suggests the lectures are perhaps complementary or targeted at the same student group.
  • Strong Negative Correlation (ϕ≈−1): Students who select L6 are highly likely to not select Lx. This suggests the lectures might be alternatives, require conflicting time slots, or appeal to entirely different student interests.
  • Weak/No Correlation (ϕ≈0): Selection of L6 has little to no impact on the selection of Lx.

Dimensionality Reduction – Clustering

For a very large number of lectures, you could use methods like Principal Component Analysis (PCA) or clustering algorithms to group similar students or lectures together. This can identify underlying student profiles (e.g., “The Data Science Crowd” or “The Humanities Enthusiasts”) that include L6 as part of their typical selection.

In this table there are students (in rows) selection of lectures (in column). My lecture is L6. I would like to know a profile of my students. so which lectures they selected the most and which the least. How strong the relations are (positive and negative if they omit some lectures.

Profile of L6 Students: Most and Least Selected Lectures

This is a Frequency Analysis of the lectures selected by the 11 students who chose L6. The percentages are based on the total number of L6 students (15).

Most Selected Lectures – The “Typical Package”: Your L6 students most frequently select L4 (67%), L3 (53%), and L5 (53%). These three form the core lecture package with L6. Least Selected Lecture: L12 is the least popular choice, selected by only 3 out of 15 students (20%).

Strength of Relation: Correlation Analysis

The Phi Coefficient quantifies the strength and direction of the relationship between choosing L6 and choosing any other lecture, using all 25 students in the dataset.

Strongest Positive Relations (Complementary): L4 (ϕ=0.263) and L5 (ϕ=0.230) show the strongest positive correlation with L6. This suggests that students interested in L6 are often those who also select L4 and L5.

Strongest Negative Relation (Alternative/Avoided): L12 (ϕ=−0.218) shows the only notable negative correlation. This confirms the frequency finding, suggesting L12 may be an alternative path or have a conflicting time/prerequisite with L6.

Weak/No Relation: Lectures like L3 and L7 have a high selection frequency but a very weak (L3) or zero (L7) correlation. This indicates that while many L6 students do take these, they are likely popular lectures chosen by many students across the board, and the choice of L6 is not a significant predictor of their selection.

Association Rule Mining

By analyzing the entire student population, we can discover general student curriculum patterns that exist beyond your specific L6 cohort. I used Association Rule Mining metrics (Support, Confidence, and Lift) to find lecture pairs that are frequently selected together.

• Support: The percentage of all 25 students who selected both lectures.
• Lift: A measure of how much the selection of one lecture increases the chance of selecting the other. A Lift>1.2 indicates a strong, meaningful positive association.

Here are the top co-selected lecture groups (pairs) among the entire student population, filtered for those selected by at least 16% of students and showing a strong positive association (Lift>1.2):

Python

import pandas as pd
from io import StringIO
import itertools

# Read the data
df = pd.read_csv(StringIO(csv_data), sep=';')
N_students = len(df)

lecture_cols = [col for col in df.columns if col.startswith('L')]
lecture_df = df[lecture_cols].fillna(0).astype(int)

# --- Association Rule Mining (Pairs) ---
# 1. Calculate Support for individual lectures
support_single = lecture_df.sum() / N_students

# 2. Calculate Support and Lift for all pairs
association_rules = []
for l1, l2 in itertools.combinations(lecture_cols, 2):
    # Calculate Support for the pair: count students who selected both
    co_selection_count = (lecture_df[l1] * lecture_df[l2]).sum()
    support_pair = co_selection_count / N_students

    # Calculate Lift: (Support(L1 and L2)) / (Support(L1) * Support(L2))
    # Handle division by zero if single support is 0, though unlikely here
    if support_single[l1] > 0 and support_single[l2] > 0:
        lift = support_pair / (support_single[l1] * support_single[l2])
    else:
        lift = 0

    # Calculate Confidence (L1 -> L2)
    confidence_l1_to_l2 = support_pair / support_single[l1] if support_single[l1] > 0 else 0

    association_rules.append({
        'Antecedent': l1,
        'Consequent': l2,
        'Support': support_pair,
        'Confidence (L1 -> L2)': confidence_l1_to_l2,
        'Lift': lift
    })

# Convert to DataFrame
rules_df = pd.DataFrame(association_rules)

# Filter for meaningful associations:
# 1. Minimum Support: Selected by at least 4 students (4/25 = 0.16)
# 2. Lift > 1.2: A strong positive relationship
min_support = 4 / N_students  # 0.16

# Filter and sort the results by Lift
top_associations = rules_df[
    (rules_df['Support'] >= min_support) &
    (rules_df['Lift'] > 1.2)
].sort_values(by='Lift', ascending=False).reset_index(drop=True)

# Add the reverse rules (L2 -> L1) to the table where Lift is high.
# Since Lift is symmetrical, only one direction needs to be calculated, but Confidence is not.

# Helper function to get Confidence (L2 -> L1) for presentation
def get_confidence_l2_to_l1(row):
    l1 = row['Antecedent']
    l2 = row['Consequent']
    support_pair = row['Support']
    return support_pair / support_single[l2] if support_single[l2] > 0 else 0

top_associations['Confidence (L2 -> L1)'] = top_associations.apply(get_confidence_l2_to_l1, axis=1)

# Reorder columns for presentation
top_associations = top_associations[['Antecedent', 'Consequent', 'Support', 'Confidence (L1 -> L2)', 'Confidence (L2 -> L1)', 'Lift']]

print("Top Co-Selected Lecture Groups (Pairs):")
print(top_associations.to_markdown(index=False, floatfmt=".3f"))

import pandas as pd
from io import StringIO
import itertools

# Read the data
df = pd.read_csv(StringIO(csv_data), sep=';')
N_students = len(df)

lecture_cols = [col for col in df.columns if col.startswith('L')]
lecture_df = df[lecture_cols].fillna(0).astype(int)

# --- Association Rule Mining (Pairs) ---
# 1. Calculate Support for individual lectures
support_single = lecture_df.sum() / N_students

# 2. Calculate Support and Lift for all pairs
association_rules = []
for l1, l2 in itertools.combinations(lecture_cols, 2):
    # Calculate Support for the pair: count students who selected both
    co_selection_count = (lecture_df[l1] * lecture_df[l2]).sum()
    support_pair = co_selection_count / N_students

    # Calculate Lift: (Support(L1 and L2)) / (Support(L1) * Support(L2))
    # Handle division by zero if single support is 0, though unlikely here
    if support_single[l1] > 0 and support_single[l2] > 0:
        lift = support_pair / (support_single[l1] * support_single[l2])
    else:
        lift = 0

    # Calculate Confidence (L1 -> L2)
    confidence_l1_to_l2 = support_pair / support_single[l1] if support_single[l1] > 0 else 0

    association_rules.append({
        'Antecedent': l1,
        'Consequent': l2,
        'Support': support_pair,
        'Confidence (L1 -> L2)': confidence_l1_to_l2,
        'Lift': lift
    })

# Convert to DataFrame
rules_df = pd.DataFrame(association_rules)

# Filter for meaningful associations:
# 1. Minimum Support: Selected by at least 4 students (4/25 = 0.16)
# 2. Lift > 1.2: A strong positive relationship
min_support = 4 / N_students  # 0.16

# Filter and sort the results by Lift
top_associations = rules_df[
    (rules_df['Support'] >= min_support) &
    (rules_df['Lift'] > 1.2)
].sort_values(by='Lift', ascending=False).reset_index(drop=True)

# Add the reverse rules (L2 -> L1) to the table where Lift is high.
# Since Lift is symmetrical, only one direction needs to be calculated, but Confidence is not.

# Helper function to get Confidence (L2 -> L1) for presentation
def get_confidence_l2_to_l1(row):
    l1 = row['Antecedent']
    l2 = row['Consequent']
    support_pair = row['Support']
    return support_pair / support_single[l2] if support_single[l2] > 0 else 0

top_associations['Confidence (L2 -> L1)'] = top_associations.apply(get_confidence_l2_to_l1, axis=1)

# Reorder columns for presentation
top_associations = top_associations[['Antecedent', 'Consequent', 'Support', 'Confidence (L1 -> L2)', 'Confidence (L2 -> L1)', 'Lift']]

print("Top Co-Selected Lecture Groups (Pairs):")
print(top_associations.to_markdown(index=False, floatfmt=".3f"))

Interpretation of Lecture Groups

The Lift values indicate the strength of the relationships:

The Strongest Cohort (The 2.2+ Lift Groups)

These are the three strongest, non-obvious combinations. A Lift≈2.2 means that students who take one lecture are more than twice as likely to take the associated lecture compared to the general student population.

• L9 ↔ L12 ↔ L8: These three lectures form a tight cluster. Specifically, L8 → L9 and L9 → L12 have extremely high association scores. This suggests a dedicated academic track or a highly correlated set of topics.

High Confidence Groups (The 1.5+ Lift Groups)

These are groups where the selection of one lecture is a very strong predictor for the other:

• L2 → L4 (87.5% Confidence): If a student selects L2, there is a very high, 87.5% chance they also select L4. This suggests L2 might be a prerequisite, a foundational course, or a direct complement to L4.
• L2 → L3 (75.0% Confidence): Similarly, L2 and L3 are frequently chosen together, indicating a strong connection.
• L1 ↔ L2: These two lectures are highly associated, suggesting they are often taken in tandem.

Does this mean that I should go to lecturers L5 and L2 and ask them for promotion of my L6, because it is most likely they find satisfactory experience on my lecture?

Data-Driven Strategy for L6 Promotion

1. Prioritize L5 (High Co-Selection & High Confidence)

You should absolutely focus on the L5 lecturer.

• L5 → L6 Confidence: 72.7%
  • This means nearly 3 out of every 4 students who take L5 also choose your L6 lecture.
  • Action: The L5 lecturer is teaching the same target audience as you. You could ask them to mention L6 as a natural follow-up or complementary course to their students.

2. Prioritize L2 (High Predictive Power)

The relationship with L2 is even more predictive of a student landing in your lecture.

• L2 → L6 Confidence: 75.0%
  • This means 3 out of every 4 students who take L2 end up in L6.
  • Action: The L2 lecturer is essentially teaching a foundational course for a majority of your class. Ask them to promote L6 as the direct next step or most relevant application course for their content.

3. The Importance of Satisfaction (What the Data Doesn’t Say)

The data confirms a strong selection link, but not a satisfaction link.

• Selection: L5 and L2 students are your target market.
• Satisfaction: To confirm they find your lecture satisfactory, you’d need student feedback/evaluation data. A student who disliked L6 might still be highly likely to take it if it’s a required course for a specific program, for example.

The strongest rationale for promotion is simply the high overlap: you are addressing a student cohort that already has a demonstrated interest pattern (L5/L2 → L6).

Secondary Focus: L4 and L3

While L4 and L3 have high total co-selection with L6 (L4: 66.7%, L3: 53.3%), their Lift and correlation scores are weaker. This means they are generally popular lectures, and while many of your students take them, the choice of L4 or L3 does not strongly predict the choice of L6.

Focusing your limited promotional time on the lecturers for L5 and L2 will yield the highest return.

Country and University as moderating factors

The factors of Country and University do show clear patterns in lecture selection among your L6 students, suggesting that existing relationships or shared academic paths likely influence their choices.

Here is the analysis of the moderating factors, based on the 15 students in your L6 lecture:

Analyzing the two largest university groups shows even sharper differences, which is expected as they are likely organized groups of students who know each other.

Theoretical Background for Moderating Factors

The Influence of Country: Cultural and Institutional Homophily

The tendency for students from the same country (e.g., Spain or Morocco) to share similar lecture profiles can be explained by Homophily and Institutional Alignment.

The Influence of University/Peer Group: Social Network Effects

The even stronger, more granular influence of the specific university groups (like the unique L7 selection by U. Malaga students) is supported by Social Influence Theory and Bounded Rationality.

By applying these concepts, you can elevate your analysis from mere correlation to a discussion of student decision-making under conditions of uncertainty and social constraint.

The lecture title informational role

Assuming that students red only the lecture titles can you analyze the whole population and discover which lectures students select as a group (l1 & l2 … ect.) based on title informational role.

This is an excellent way to use the data to understand the thematic logic that guides students’ decisions, rather than just the frequency. By analyzing the corpus of words in the high-Lift lecture titles, we can discover the “driving wording” that creates strong, predictable co-selection patterns.

The primary driving themes are Economics, Management, and Business, but the strongest associations occur when these words are combined with qualifying terms like International, Applied, or Process.

Analysis of Driving Wording in Lecture Titles

Based on the highest Lift scores (strongest association patterns), the lecture groups cluster into four distinct thematic tracks driven by specific keywords:

1. Driving Theme: International & Political Economy 🌍

This is the strongest thematic driver in the entire dataset, creating three of the top four co-selection groups.

Driving Wording: International, Economy, Political.

2. Driving Theme: Applied Economic Analysis

This theme links foundational economic knowledge with quantitative skills and real-world application.

Driving Wording: Applied, Analysis, Economics, Decisions.

3. Driving Theme: Business Management & Strategy

This group is driven by a focus on business processes and the organizational changes brought by technology.

Driving Wording: Management, Process, Digital, Innovation.

Summary of Driving Wording

The strongest selection drivers are not single words but rather qualified thematic phrases that define an academic specialization:

Modern Business Specialization: Driven by Management combined with Process or Digital/Innovation (L5, L6, L7, L11).

Macro/Global Specialization: Driven by International and Political Economy (L9, L12, L8).

Quantitative Specialization: Driven by Applied Economics and Business Analytics (L2, L3, L1).

About wording analysis you can read here: Organizational aspiration for social impact

What recommendation to lecture L6 in wording. How can I modify or extend title of L6 to experiment for more students in next period.

The goal is to maximize enrollment by aligning the L6 title with the thematic drivers of the most predictive lectures (L5, L2, L4). Your current title is L6: Customer Experience Management.

Here are the recommendations for wording modifications, structured by the target student profile you want to attract:

Recommended Title Modifications for L6

Strategy 1: Attract L5 Students (Strongest Existing Link)

The L5 title is Business process management. These students seek internal efficiency as a foundation for external success.

Strategy 2: Attract L2/L4 Students (Applied & Analytical)

The L2 title is Applied Economics Real-World Challenges and Solutions. L4 is Business plan. These students are practical and analytical.

Strategy 3: Attract L7/L11 Students (Future/Digital Focus)

The L7/L11 titles feature Innovation and Digital Economy.

Final Recommendation: The Best Title to Test

The most balanced and powerful title that leverages multiple co-selection themes is: Customer Experience Management: Data, Process, and Digital Strategy

This phrase:

• Includes Management (L5, L6, L11 theme).
• Includes Process (L5 link).
• Includes Data (L2/L3 Analytics link).
• Includes Digital Strategy (L7/L11 Innovation link).

Thank you for reading.

The post Student Profiling for your lecture appeared first on Customer Experience Management.

The symmetric data analysis (e.g., correlation and multiple regression analysis) …

The asymmetric data analysis (i.e., individual case outcome forecasts) …

Based on: Fuzzy-set Qualitative Comparative Analysis (fsQCA): Guidelines for research practice in Information Systems and marketing (Ilias O. Pappas, Arch G. Woodside)

Qualitative inductive reasoning with data being analyzed “by case’’ and not “by variable’’, is combined with quantitative empirical testing, as sufficient and necessary conditions identify outcomes through statistical methods. In most cases, QCA are useful in quantitative studies, as it allows the researcher to get a deep view of their data through a quantitative analysis. That has also several characteristics of qualitative analysis.

Case studies focus on describing, explaining, and forecasting, single and combinatorial conditional antecedents on outcomes while variable studies focus on the similarities of variances of two or more variables. A “condition” is a point or interval range of antecedent or outcome; a “variable” characteristic varies.

Here are few examples of conditions versus variables: “Male” is a condition; “gender” is a variable. “Swedish” is a condition; “nationality” is a variable. “Expert” is a condition; “expertise” is a variable.

The goal of QCA is to explain causality in complex real life phenomena. QCA goes through “multiple-conjunctural causation, which refers to “nonlinear, nonadditive, non-probabilistic conception that rejects any form of permanent causality. That stresses different paths which can lead to the same outcome. QCA investigate complex combinations of conditions and diversity. QCA uses Boolean algebra and Boolean minimization algorithms to capture patterns of multiple-conjunctural causation and to simplify complex data structures.

Types qualitative comparative analysis (QCA)

CsQCA and mvQCA

CsQCA is the first variation of QCA. It is a tool created to deal with complex sets of binary data. The use of Boolean algebra means that QCA has as input binary data (0 or 1). That make QCA uses logical operations for the procedure. Thus it is very important to dichotomize the use of variables in a useful and meaningful manner.

mvQCA, treats variables as multi-valued instead of dichotomous. MvQCA retains the idea of performing a synthesis of the dataset and cases with the same value on the outcome variable. They are explained by a solution, which contains combinations of variables that explain a number of cases with the outcome.

FsQCA addresses an important limitation of csQCA, the fact that variables are binary, thus restricting the analysis as it cannot fully capture the complexity in cases that naturally vary by level or degree. This restriction of csQCA is likely an important reason that QCA has not been widely adopted in multiple contexts, including IS and marketing research. FsQCA extends csQCA by integrating fuzzy-sets and fuzzy-logic principles with QCA. The variables can get all the values within the range of 0–1. FsQCA is able to overcome several limitations of both csQCA and mvQCA, and has received increased attention recently. FsQCA applies together with complexity theory, it provides the opportunity to gain deeper and richer insight into data.

FsQCA and cluster analysis

Case-based techniques, such as fsQCA and cluster analysis, have been employed as a way of moving beyond variance-based methods. These two techniques have similarities as they both employ multidimensional spaces and often people ask how fsQCA differs from cluster analysis and why do we need it. A main difference between the two methods is the kind of research questions they are able to address.

Specifically, cluster analysis answers questions such as which cases are more similar to each other, while fsQCA can identify the different configurations that constitute sufficient and/or necessary conditions for the outcome of interest. Depending on the focus of the study the researcher should choose the most appropriate method. Their differences stem from the fact that “QCA addresses the positioning of cases in [multidimensional] spaces via set theoretic operations while cluster analysis relies on geometric distance measures and concepts of variance minimization” . To this end, prior studies compare fsQCA with cluster analysis and show how fsQCA can handle causal complexity with fine-grained level data, or how it can identify more solutions compared to cluster analysis. A discussion exists in the literature regarding QCA and cluster analysis, and both approaches have differences making them suitable for different types of studies.

read example here: Hierarchical Agglomerative Clustering for Product Grouping

Adoption of fsQCA in relevant studies

Configurational approaches are becoming more popular over the past few years in different areas, with fsQCA playing a large part in this as most studies will prefer fuzzy-set over crisp-set and multi-value QCA (Thiem & Dusa, 2013). In detail, fsQCA has been employed in information systems (Fedorowicz, Sawyer, & Tomasino, 2018; Liu et al., 2017), online business and marketing (Pappas et al., 2016; Pappas, 2018; Woodside, 2017), consumer psychology (Schmitt, Grawe, & Woodside, 2017), strategy and organizational research (Fiss, 2011; Greckhamer et al., 2018), education (Pappas, Giannakos et al., 2017; Plewa, Ho, Conduit, & Karpen, 2016), data science (Vatrapu, Mukkamala, Hussain, & Flesch, 2016) and learning analytics (Papamitsiou et al., 2018; Sergis, Sampson, & Giannakos, 2018). This tutorial aims to increase the adoption of fsQCA in IS and marketing studies following the call for more empirical work in the area (El Sawy, Malhotra, Park, & Pavlou, 2010; Fiss, 2011; Woodside, 2014, 2017).

Adoption of fsQCA in relevant studies

FsQCA is useful for both inductive and deductive reasoning for theory building, elaboration, and testing. This analysis allows to identify specific cases in the sample. With this knowledge, the researcher can go back to the cases and use contextual information, not including in the analysis, to further explain and discuss the findings.

A typical variance-based analysis would identify a single best solution, thus limiting the results. FsQCA studies can compare the findings between different data analysis techniques to describe how different stories are hidden in the same dataset. It is recommended to combine fsQCA with other data analysis techniques if possible.

How to use fsQCA in a typical e-commerce study

Sampling

The study examined cognitive and affective perceptions as antecedents of online shopping behavior in personalized e-commerce environments. We used a typical a snowball sampling methodology to recruit participants and controlled for respondents’ previous experience with both online shopping and personalized services. Eventually, the sample comprises 582 individuals with experience in online shopping and personalized services. We collected data through a questionnaire built with measures that have been adopted from the literature. Appendix A (as presented in the original study) lists construct definitions, the questionnaire items used to measure each construct, along with descriptive statistics and loadings.

Evaluate constructs for reliability and validity

Typical with similar quantitative studies, first we evaluate constructs for reliability and validity. This is a step that should be always performed when it is appropriate, and it is not directly related with the fsQCA analysis as it depends on the type of variables that are used in the study. Construct reliability and validify, as the name implies, refer to the construct itself and not to the method of analysis used to examine relations between constructs.

Reliability testing, based on the Cronbach alpha indicator, showed acceptable indices of internal consistency since all constructs exceed the cut-off threshold of 0.70. The AVE for all constructs ranged between 0.55 and 0.84, all correlations were lower than 0.80, and square root AVEs for all constructs were larger than their correlations. The findings in detail for the confirmatory analysis may be found in the original paper.

Contrarian case analysis

Contrarian case analysis is performed outside fsQCA, but we present it here because it can serve as an easy and quick way to examine how many cases in our sample are not explained by main effects, and thus they would not be included in the outcome of a typical variance-based approach, e.g., correlation or regression analysis.

Data Calibration

In fsQCA, different from traditional methods, instead of working with probabilities data are transformed from ordinal or interval scales into degrees of membership in the target set, which shows if and how much a case belongs into a specific set. “In essence, a fuzzy membership score attaches a truth value, not a probability, to a statement”.

For example, the variable intention to purchase can be coded as “high intention to purchase”, and we will be looking for the presence or absence of the condition high intention to purchase (“intention to purchase” is the variable; “high intention to purchase” is a condition). Similarly, we code the rest of the variables.

The method computes the presence of a condition or its opposite (i.e., negation). The negation of a condition is referred in the literature as the absence of a condition, and the two terms have been used interchangeably based on how the absence is computed. The term absence has been also used to describe when the condition is irrelevant in a configuration. It is similar to the “do not care” term that is also often used in the literature.

This distinction is not often addressed or clarified, thus we suggest researchers to clearly define these terms in future works to avoid misunderstandings.

Transform data into fuzzy-sets

In fsQCA we need to calibrate our variables to form fuzzy sets with their values ranging from 0 to 1. Consider a fuzzy set as a group, then the values from 0 to 1 define if and at what amount a case belongs to this group. The fact that all values range from 0 to 1 means that a case with a fuzzy membership score of 1 is a full member of a fuzzy set (fully in the set), and a case with a membership score of 0 is a full non-member of the set (fully out of the set). A membership score of 0.5 is exactly in the middle, thus a case would be both a member of the fuzzy set and a non-member, and is therefore a member of what is known as the intermediate set. The intermediate-set point is the value where there is maximum ambiguity as to whether a case is more in or more out of the target set.

Data calibration may be either direct or indirect. In the direct calibration the researcher needs to choose exactly three qualitative breakpoints, which define the level of membership in the fuzzy set for each case (fully in, intermediate, fully out). In the indirect method, the measurements need to be rescaled based on qualitative assessments. The researcher may choose to calibrate a measure differently, depending on what one is investigating. Either method may be chosen, depending on researcher’s substantive knowledge of both data and underlying theory. The direct method is recommended and is more common, in which the researcher sets three values corresponding to full-set membership, full-set non-membership, and intermediate-set membership. This can lead to more rigorous studies which are easier to be replicated and validated, since it is clearer on how the thresholds have been chosen.

The percentiles allow the calibration of any measure regardless of its original values. In detail, we can compute the 95 %, 50 %, and 5 % of our measures and use these values as the three thresholds in fsQCA software.

Especially in the case of the widely used seven-point Likert scales (1=Not at all, 7=Very much), previous studies suggest that the values of 6, 4, and 2 can be used as thresholds. Similarly, for a five-point Likert scale the thresholds could be 4,3, and 2.

Interpreting and presenting the solutions

FsQCA software provides all three solutions every time. Complex and parsimonious solutions are computed regardless of any simplifying assumptions employed by the researcher (e.g., choosing the presence or absence/negation of a variables) while the intermediate solution depends on these assumptions. While the intermediate solution includes both core and peripheral conditions, we need an easy way to make the distinction that will help us interpret and present the solutions in a better manner.

To improve the presentation of the findings we can transform the solutions from fsQCA output into a table that is easier to read. Typically,

1. the presence of a condition is indicated with a black circle (●),
2. the absence/negation with a crossed-out circle (⊗),
3. and the “do not care” condition with a blank space.

The negation of a condition is referred in the literature also as absence, and the two terms have been used interchangeably. The distinction between core and peripheral is made by using large and small circles, respectively. The researcher needs to present the overall solution consistency and the overall solution coverage. The overall coverage describes the extent to which the outcome of interest may be explained by the configurations, and is comparable with the R-square reported on regression-based methods. In our example, the results indicate an overall solution coverage of 0.84, which suggests that a substantial proportion of the outcome is covered by the nine solutions.

All graphics and futher explanation you find here: Pappas, I. O., & Woodside, A. G. (2021). Fuzzy-set Qualitative Comparative Analysis (fsQCA): Guidelines for research practice in Information Systems and marketing. International journal of information management, 58, 102310.

The post Fuzzy-Set Qualitative Comparative Analysis appeared first on Customer Experience Management.

Independent variables in simple terms:

• Independent variables: Inputs you control or observe.
• Dependent variable: Output you want to understand or predict.

Example of independent variables in Customer Management (RFM Model):

Suppose you’re analyzing customer behavior to predict churn (whether a customer will stop buying).

• Independent variables:
  • Recency: How recently a customer made a purchase.
  • Frequency: How often they purchase.
  • Monetary: How much they spend.
• Dependent variable:
  • Churn: 1 if the customer churned, 0 if they stayed.

In this case, Recency, Frequency, and Monetary are independent variables used to predict the likelihood of churn. See also Python for business analytics – rfm analysis

Why check for independence among independent variables?

If independent variables are highly correlated with other variables, i.e., it is not truly independent, it can cause multicollinearity, which makes model coefficients unstable, reduces interpretability, and can lead to misleading conclusions.

Multicollinearity

Multicollinearity refers to a statistical phenomenon in which two or more independent variables in a regression model are highly correlated. This makes it difficult to determine the individual effect of each variable on the dependent variable because they essentially carry overlapping information.

Assessment of Multicollinearity

To assess multicollinearity, you can use following methods:

1. Correlation Matrix
   • Check pairwise correlations between independent variables.
   • High correlation (e.g., > 0.8 or < -0.8) may indicate multicollinearity.
2. Variance Inflation Factor (VIF)
   • Measures how much the variance of a regression coefficient is inflated due to multicollinearity.
   • VIF > 5 or 10 is often considered problematic.
3. Tolerance
   • Tolerance = 1 / VIF.
   • Low tolerance values (close to 0) indicate high multicollinearity.
4. Condition Index and Eigenvalues
   • Part of a more advanced diagnostic using matrix decomposition.
   • A condition index > 30 may suggest serious multicollinearity.

How to deal with multicollinearity?

• Remove one of the correlated variables.
• Combine variables (e.g., using PCA or creating an index).
• Regularization techniques like Ridge or Lasso regression.
• Centering variables (subtracting the mean) can help in some cases.

Calculation example

Assume, you have data similar to this sample.

Variable independence testing

import pandas as pd

# Load the data with the specified delimiter
df = pd.read_csv("RFM_analysis_614.csv", delimiter=",")

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Select the independent variables (RFM)
X = df[['recency', 'freq', 'monetary']]

# Add a constant for the VIF calculation (required by the statsmodels function)
X = add_constant(X)

# Create a DataFrame to hold the VIF results
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

# Exclude the constant row from the final output since it's not a true variable
vif_data = vif_data[vif_data.Variable != 'const'].reset_index(drop=True)

print(vif_data)

# Save the VIF results to a CSV file
vif_data.to_csv("vif_results.csv", index=False)

import pandas as pd

# Load the data with the specified delimiter
df = pd.read_csv("RFM_analysis_614.csv", delimiter=",")

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Select the independent variables (RFM)
X = df[['recency', 'freq', 'monetary']]

# Add a constant for the VIF calculation (required by the statsmodels function)
X = add_constant(X)

# Create a DataFrame to hold the VIF results
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

# Exclude the constant row from the final output since it's not a true variable
vif_data = vif_data[vif_data.Variable != 'const'].reset_index(drop=True)

print(vif_data)

# Save the VIF results to a CSV file
vif_data.to_csv("vif_results.csv", index=False)

Finally program prints following results

Based on the Variance Inflation Factor (VIF) calculation, the columns recency, frequency, and monetary are statistically independent of each other and not informationally overlapping.

This means that you can use all three variables together as independent predictors in a statistical model, such as a Cox Proportional Hazards (Cox PH) model, without concern for severe multicollinearity.

Variance Inflation Factor (VIF) Results

The VIF (https://en.wikipedia.org/wiki/Variance_inflation_factor) is a measure of how much the variance of an estimated regression coefficient is increased due to collinearity. A common rule of thumb is that a VIF value less than 5 or sometimes 10 indicates that the correlation between the variables is not high enough to warrant concern.

The calculated VIF values for your RFM variables are very low:

Conclusion on Independence

Since all VIF values are close to 1.0 and well below the 5.0 threshold:

• Independent Variables: You can confidently treat recency, frequency, and monetary as independent variables for your statistical analysis (e.g., in a Cox PH model).
• No Informational Overlap: The variables are providing distinct, non-redundant information to the model. For instance, knowing a customer’s frequency does not allow the model to strongly predict their recency or monetary value.

The post What are independent variables? appeared first on Customer Experience Management.

1. PyCharm

• Developer: JetBrains
• Platforms: Windows, macOS, Linux
• Features: Advanced code analysis, smart code completion, integrated debugger, Git support, virtual environment management, Django support.
• Community (free) and Professional (paid) editions.
• Website: PyCharm

2. Visual Studio Code (VS Code)

• Developer: Microsoft
• Platforms: Windows, macOS, Linux
• Features: Lightweight but powerful; excellent Python extension; integrated terminal; rich plugin ecosystem; Git support; Jupyter notebook integration.
• Website: VS Code

3. Spyder

• Developer: Scientific Python Development Environment Community
• Platforms: Windows, macOS, Linux
• Features: Focused on scientific computing and data science; variable explorer; integrated IPython console; plotting support.
• Website: Spyder

4. Thonny

• Developer: University of Tartu
• Platforms: Windows, macOS, Linux
• Features: Beginner-friendly; simple UI; built-in debugger; good for learning and education.
• Website: Thonny

5. Wing IDE

• Developer: Wingware
• Platforms: Windows, macOS, Linux
• Features: Powerful debugger; code intelligence; remote development support.
• Website: Wing IDE

6. Eric

• Developer: Detlev Offenbach
• Platforms: Windows, macOS, Linux
• Features: Full-featured Python and Ruby IDE; integrated debugger; plugin support.
• Website: Eric Python IDE

7. IDLE

• Developer: Python Software Foundation (bundled with Python)
• Platforms: Windows, macOS, Linux
• Features: Basic, lightweight, good for quick scripts and learning.
• Website: IDLE Documentation

Summary:
For professional development, PyCharm and VS Code are the most popular. For data science, Spyder is widely used. For beginners, Thonny or IDLE are great choices.

The post Top IDEs for Python developers appeared first on Customer Experience Management.

In this tutorial, we’ll show you exactly how to do that. We’ll build a powerful yet simple Python script that automatically scans a webpage, finds all the PDF links, and searches for specific text inside each one. Using popular libraries like Requests, BeautifulSoup, and PyPDF, you’ll learn a practical skill that can save you hours of manual work. Let’s get started!

Python, Web Scraping, PDF, Automation, BeautifulSoup, PyPDF, requests, Data Extraction, Python Projects, Text Search

Scrape a Website

in the script we Find all links on the initial page. Filter for links that end with .pdf. For each PDF link: Download the PDF file into memory. Extract text from every page of the PDF. Search the extracted text for your search_string. And finally Report which PDF files contain the phrase. Scrape a Website

import requests
from bs4 import BeautifulSoup
from pypdf import PdfReader
import io

Scrape a Website
def find_linked_pdfs(url):
    """
    Scans a webpage for PDF links and searches for a string within each PDF.

    Args:
        url: The URL of the webpage to scan.
        search_string: The string to search for inside the PDFs.
    """
    print(f"Scanning {url} for PDF links...")
    try:
        # 1. Get the main page to find all links
        base_url_parts = requests.utils.urlparse(url)
        base_url = f"{base_url_parts.scheme}://{base_url_parts.netloc}"
        
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')

        pdf_links = [a['href'] \
          for a in soup.find_all('a', href=True) \
          if a['href'].endswith('.pdf')]
        
        if not pdf_links:
            print("No PDF links found on the page.")
            return

        print(f"Found {len(pdf_links)} PDF files. Now searching inside them...")

    return pdf_links

import requests
from bs4 import BeautifulSoup
from pypdf import PdfReader
import io

Scrape a Website
def find_linked_pdfs(url):
    """
    Scans a webpage for PDF links and searches for a string within each PDF.

    Args:
        url: The URL of the webpage to scan.
        search_string: The string to search for inside the PDFs.
    """
    print(f"Scanning {url} for PDF links...")
    try:
        # 1. Get the main page to find all links
        base_url_parts = requests.utils.urlparse(url)
        base_url = f"{base_url_parts.scheme}://{base_url_parts.netloc}"
        
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')

        pdf_links = [a['href'] \
          for a in soup.find_all('a', href=True) \
          if a['href'].endswith('.pdf')]
        
        if not pdf_links:
            print("No PDF links found on the page.")
            return

        print(f"Found {len(pdf_links)} PDF files. Now searching inside them...")

    return pdf_links

Handling URLs: It constructs a full, absolute URL for each PDF, as many links on a page can be relative (e.g., /path/to/file.pdf). Scrape a Website. https://pypi.org/project/beautifulsoup4/

In-Memory Processing: Instead of saving each PDF to your disk, it uses io.BytesIO to treat the downloaded content as a file in your computer’s memory. This is faster and cleaner.

Text Extraction: The pypdf library’s PdfReader opens this in-memory file. The script then loops through each page, calls extract_text(), and combines the text from all pages.

Searching and Reporting: Finally, it performs a case-insensitive search on the extracted text and prints the URL of any PDF that contains your search term.

Search Inside PDF
def find_text_in_pdfs(pdf_links, search_string):

        # 2. Loop through each PDF link

        found_in_files = []
        for pdf_path in pdf_links:
            # Construct absolute URL if the link is relative
            if not pdf_path.startswith(('http://', 'https://')):
                pdf_url = f"{base_url}{pdf_path}"
            else:
                pdf_url = pdf_path

            try:
                # 3. Download the PDF content
                pdf_response = requests.get(pdf_url)
                pdf_response.raise_for_status()

                # Use an in-memory buffer to read the PDF
                pdf_file = io.BytesIO(pdf_response.content)
                reader = PdfReader(pdf_file)
                
                # 4. Extract text and search
                full_text = ""
                for page in reader.pages:
                    full_text += page.extract_text() or ""
                
                if search_string.lower() in full_text.lower():
                    print(f"✔️ Found '{search_string}' in: {pdf_url}")
                    found_in_files.append(pdf_url)

            except Exception as e:
                print(f"⚠️ Could not process {pdf_url}. Reason: {e}")
        
        if not found_in_files:
            print(f"\nSearch complete. The string '{search_string}' was not found in any of the PDFs.")

    except requests.exceptions.RequestException as e:
        print(f"An error occurred fetching the main URL: {e}")

Search Inside PDF
def find_text_in_pdfs(pdf_links, search_string):

        # 2. Loop through each PDF link

        found_in_files = []
        for pdf_path in pdf_links:
            # Construct absolute URL if the link is relative
            if not pdf_path.startswith(('http://', 'https://')):
                pdf_url = f"{base_url}{pdf_path}"
            else:
                pdf_url = pdf_path

            try:
                # 3. Download the PDF content
                pdf_response = requests.get(pdf_url)
                pdf_response.raise_for_status()

                # Use an in-memory buffer to read the PDF
                pdf_file = io.BytesIO(pdf_response.content)
                reader = PdfReader(pdf_file)
                
                # 4. Extract text and search
                full_text = ""
                for page in reader.pages:
                    full_text += page.extract_text() or ""
                
                if search_string.lower() in full_text.lower():
                    print(f"✔️ Found '{search_string}' in: {pdf_url}")
                    found_in_files.append(pdf_url)

            except Exception as e:
                print(f"⚠️ Could not process {pdf_url}. Reason: {e}")
        
        if not found_in_files:
            print(f"\nSearch complete. The string '{search_string}' was not found in any of the PDFs.")

    except requests.exceptions.RequestException as e:
        print(f"An error occurred fetching the main URL: {e}")

x

if __name__ == "__main__":
    target_url = "https://www.umcs.pl/pl/plany-zajec,10795.htm"
    search_term = "programming"
    pdf_links = find_linked_pdfs(target_url)
    find_text_in_pdfs(pdf_links, search_string)
    

if __name__ == "__main__":
    target_url = "https://www.umcs.pl/pl/plany-zajec,10795.htm"
    search_term = "programming"
    pdf_links = find_linked_pdfs(target_url)
    find_text_in_pdfs(pdf_links, search_string)
    

Wrapping Up and Next Steps

Congratulations! You’ve successfully built a powerful automation script that bridges the gap between web scraping and document analysis. By combining the strengths of Requests, BeautifulSoup, and PyPDF, you can now programmatically find information that was previously locked away inside PDF files on any website. This not only saves an incredible amount of time but also opens up new possibilities for data collection and analysis. Feel free to adapt the code for your own projects and take your web scraping skills to the next level. Scrape a Website.

The applications for this technique extend far beyond a single use case. Imagine using this script for academic research, automatically scanning university archives for papers mentioning a specific topic. You could adapt it for financial analysis by pulling keywords from dozens of quarterly earnings reports, or for legal work by searching through court filings for a particular case name. Job seekers could even use it to scan company websites for PDF job descriptions that contain key skills.

To perform a statistical analysis of the overall economy, you can leverage a variety of online resources, including government and intergovernmental data portals, as well as academic publications. These sources often provide data in structured formats like CSVs and APIs, but also in less-structured formats like HTML tables and PDFs, which can be parsed using Python libraries like Beautiful Soup and pypdf.

Government and Intergovernmental Data Sources

For raw, official economic data, these are your most reliable sources. They offer a wealth of information on everything from GDP and inflation to employment rates and international trade. Scrape a Website. Search Inside PDF

• Federal Reserve Economic Data (FRED): A fantastic resource from the St. Louis Fed, FRED offers over 800,000 economic time series from more than 100 sources. It’s a goldmine for anyone doing macroeconomic analysis.
• The World Bank Open Data: This portal provides comprehensive global development data, including indicators on economic policy, poverty, gender, and more, making it perfect for cross-country comparisons.
• Data.gov: The home of U.S. government open data, this site aggregates datasets from various federal agencies, including the Bureau of Economic Analysis (BEA) and the Bureau of Labor Statistics (BLS).
• United Nations Statistics Division (UNSD): The UNSD offers a wide array of international statistics, including the UNdata portal which provides free access to over 60 million statistical records from various UN agencies.
• The Bureau of Economic Analysis (BEA): The BEA produces some of the most critical U.S. economic statistics, such as GDP, personal income, and corporate profits.

You can read about Business analyst carrier path here

The core principle remains the same: automate the discovery of information, no matter the format. Search Inside PDF. Happy coding! 🚀

The post How to Scrape a Website and Search Inside PDFs with Python appeared first on Customer Experience Management.

Vector vs lexical search

An easy way to introduce vector search is by comparing it to the more conventional lexical search that you’re probably used to. Vector search, also commonly known as semantic search, and lexical search work very differently.

Lexical search is the kind of search that we’ve all been using for years. To summarize it very briefly, it doesn’t try to understand the real meaning of what is indexed and queried, instead, it makes a big effort to lexically match the literals of the words or variants of them like stemming words, or synonyms, etc.. That makes what the user types in a query with all the literals that have been previously indexed into the database. The similarity is replaced by ranking algorithm, such as TF-IDF.

Documents are tokenized and analyzed. Then, the resulting terms are indexed in an inverted index, which simply maps the analyzed terms to the documents containing them. Searching for “yellow texas roses” will match all documents with varying scores.

Semantic search – the whole purpose of semantic search is to index data in such a way that it can be searched based on the meaning it represents.

What’s the difference between semantic search and lexical search?

Lexical search doesn’t try to understand the real meaning of what is indexed and queried- it matches the literals of the words or their variants. In contrast, vector search indexes data in a way that allows it to be searched based on the meaning it represents.

Read more about vector similarity https://www.elastic.co/search-labs/blog/introduction-to-vector-search

Semantic search with elasticsearch

Load model. See details here and here.

from elasticsearch import Elasticsearch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-base")
VECTOR_DIMENSION = model.get_sentence_embedding_dimension()
print(f"Model loaded successfully.VECTOR_DIMENSION {VECTOR_DIMENSION}")

from elasticsearch import Elasticsearch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-base")
VECTOR_DIMENSION = model.get_sentence_embedding_dimension()
print(f"Model loaded successfully.VECTOR_DIMENSION {VECTOR_DIMENSION}")

Start elastic server

from elasticsearch import Elasticsearch
ES_HOST = "http://localhost:9000"
es = Elasticsearch(hosts=[ES_HOST])
print(f"Connection successful: {es.info().body['cluster_name']}")

from elasticsearch import Elasticsearch
ES_HOST = "http://localhost:9000"
es = Elasticsearch(hosts=[ES_HOST])
print(f"Connection successful: {es.info().body['cluster_name']}")

Index product data to elastic database

# Fetch products from database
def select_products():
    q = """
        SELECT *
        FROM [DB_Products] 
        where [BrandName] in ('PCE','De Walt')
        """
    dfp = read_from_sql_server(q, odbc_conect='DSN=SQLxx')
    return dfp

# >-------------------------------------------------
df_docs = select_products()
print(df_docs.info())
# -------------------------------------------------

# Fetch products from database
def select_products():
    q = """
        SELECT *
        FROM [DB_Products] 
        where [BrandName] in ('PCE','De Walt')
        """
    dfp = read_from_sql_server(q, odbc_conect='DSN=SQLxx')
    return dfp

# >-------------------------------------------------
df_docs = select_products()
print(df_docs.info())
# -------------------------------------------------

# Index product names into elastic
index_mapping = {
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims":  VECTOR_DIMENSION,
                "index": True,
                "similarity": "cosine"
                },

            "product_name": { 
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256 
                    }
                  }
                },
            "product_name_org": {
                "type": "keyword",
                "ignore_above": 256
                },
            "prodidx": {
                "type": "keyword",
                "ignore_above": 256
                },
            "img_url": {
                "type": "keyword",
                "ignore_above": 512 
                },
            }
        }

# --- 4. Create the Index -------------------------------------------------

INDEX_NAME = 'prod_names_for_search_hybrid'

import time
es.options(ignore_status=[400, 404]).indices.delete(index=INDEX_NAME, ignore_unavailable=True)
time.sleep(3) # Give a moment for index deletion to propagate

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, mappings=index_mapping)
    print("Index created.")
else:
    print("Index already exists.")
# -----------------------------------------------------------------------

# Indexing product data ------------------------------------------------- 
product_names = df_docs['ProductName'].apply(lambda x: x.lower()[:256]).to_list()

i = 1
for name in product_names:
    print(f"Indexing:{i} {name}")
    i += 1
    
    vector = model.encode(f"passage: {name}").tolist()
    doc = {
        "Product_name": name,
        "ProductNameVector": vector
    }
    
    es.index(index=INDEX_NAME, document=doc, refresh=True)

# ---------------------------------------------------------------------

# Index product names into elastic
index_mapping = {
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims":  VECTOR_DIMENSION,
                "index": True,
                "similarity": "cosine"
                },

            "product_name": { 
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256 
                    }
                  }
                },
            "product_name_org": {
                "type": "keyword",
                "ignore_above": 256
                },
            "prodidx": {
                "type": "keyword",
                "ignore_above": 256
                },
            "img_url": {
                "type": "keyword",
                "ignore_above": 512 
                },
            }
        }

# --- 4. Create the Index -------------------------------------------------

INDEX_NAME = 'prod_names_for_search_hybrid'

import time
es.options(ignore_status=[400, 404]).indices.delete(index=INDEX_NAME, ignore_unavailable=True)
time.sleep(3) # Give a moment for index deletion to propagate

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, mappings=index_mapping)
    print("Index created.")
else:
    print("Index already exists.")
# -----------------------------------------------------------------------

# Indexing product data ------------------------------------------------- 
product_names = df_docs['ProductName'].apply(lambda x: x.lower()[:256]).to_list()

i = 1
for name in product_names:
    print(f"Indexing:{i} {name}")
    i += 1
    
    vector = model.encode(f"passage: {name}").tolist()
    doc = {
        "Product_name": name,
        "ProductNameVector": vector
    }
    
    es.index(index=INDEX_NAME, document=doc, refresh=True)

# ---------------------------------------------------------------------

You can check your index using diagnostic functions. Semantic search.

# --- Diagnostic functions  ---------------------------------------
def indices_list():
    return [index['index'] for index in es.cat.indices(format='json')]   

def count_records(index_name):
    return es.count(index=index_name)['count']    
# -----------------------------------------------------------------

# Get indices and record count ------------------------------------
for i in indices_list():
    print(i, count_records(i))

# Get mapings
mapping = es.indices.get_mapping(index=INDEX_NAME)
fields = mapping[INDEX_NAME]['mappings']['properties']
for field, details in fields.items():
    print(f"Model {INDEX_NAME}, fields {field}: {details['type']}")
#------------------------------------------------------------------

# --- Diagnostic functions  ---------------------------------------
def indices_list():
    return [index['index'] for index in es.cat.indices(format='json')]   

def count_records(index_name):
    return es.count(index=index_name)['count']    
# -----------------------------------------------------------------

# Get indices and record count ------------------------------------
for i in indices_list():
    print(i, count_records(i))

# Get mapings
mapping = es.indices.get_mapping(index=INDEX_NAME)
fields = mapping[INDEX_NAME]['mappings']['properties']
for field, details in fields.items():
    print(f"Model {INDEX_NAME}, fields {field}: {details['type']}")
#------------------------------------------------------------------

Semantic search

Now when you have index you can search. You can read more in my post Semantic Search with Elasticsearch

# Semantic Search --- 

query_text = "yellow texas rose"
query_vector = model.encode(f"query: {query_text}").tolist()
knn_query = {
    "field": "embedding", #"ProductNameVector",
    "query_vector": query_vector,
    "k": 10,
    "num_candidates": 50
}
response = es.search(index=INDEX_NAME, knn=knn_query, source=["product_name"])
for hit in response['hits']['hits']:
    print(f"  - Product: {hit['_source']['product_name']} (Score: {hit['_score']:.4f})")

# Semantic Search --- 

query_text = "yellow texas rose"
query_vector = model.encode(f"query: {query_text}").tolist()
knn_query = {
    "field": "embedding", #"ProductNameVector",
    "query_vector": query_vector,
    "k": 10,
    "num_candidates": 50
}
response = es.search(index=INDEX_NAME, knn=knn_query, source=["product_name"])
for hit in response['hits']['hits']:
    print(f"  - Product: {hit['_source']['product_name']} (Score: {hit['_score']:.4f})")

You can perform also lexical search or finnali hybrid search.

def search_score_script(query_h):
    response = es.search(
        index=INDEX_NAME,
        body={
            "_source": ['product_name','prodidx','img_url'],
            "query": {
                "function_score": {
                    "query": {
                        "bool": {
                            "should": [
                                {
                                    "match": {
                                        "product_name": {
                                            "query": query_h,
                                            'operator': 'and',
                                            "_name": "text_match"
                                        }
                                    }
                                },
                                {
                                    "knn": {
                                        "field": "embedding",
                                        "query_vector": model.encode(f"query: {query_text}").tolist(),
                                        "k": 30,
                                        "num_candidates": 300,
                                        "_name": "semantic_search"
                                    }
                                }
                            ]
                        }
                    },                 
                }
            },
            "size": 100
        }
    )
    return response

response = search_score_script(query_h)   
products = [
         {
             "product_name": hit["_source"]["product_name"],
             "score": hit["_score"],
             "matched_queries": hit.get("matched_queries", []),
             "prodidx": hit["_source"]["prodidx"],
         }
         for hit in response["hits"]["hits"]
     ]

print([(p["product_name"],p["score"] for p in products])
 

def search_score_script(query_h):
    response = es.search(
        index=INDEX_NAME,
        body={
            "_source": ['product_name','prodidx','img_url'],
            "query": {
                "function_score": {
                    "query": {
                        "bool": {
                            "should": [
                                {
                                    "match": {
                                        "product_name": {
                                            "query": query_h,
                                            'operator': 'and',
                                            "_name": "text_match"
                                        }
                                    }
                                },
                                {
                                    "knn": {
                                        "field": "embedding",
                                        "query_vector": model.encode(f"query: {query_text}").tolist(),
                                        "k": 30,
                                        "num_candidates": 300,
                                        "_name": "semantic_search"
                                    }
                                }
                            ]
                        }
                    },                 
                }
            },
            "size": 100
        }
    )
    return response

response = search_score_script(query_h)   
products = [
         {
             "product_name": hit["_source"]["product_name"],
             "score": hit["_score"],
             "matched_queries": hit.get("matched_queries", []),
             "prodidx": hit["_source"]["prodidx"],
         }
         for hit in response["hits"]["hits"]
     ]

print([(p["product_name"],p["score"] for p in products])
 

Sentence Transformers (e.g., MiniLM, BERT variants) are the best choice models for semantic search

• Type: Dense vector models.
• Pros:
  • Rich semantic understanding.
  • Multilingual support.
  • Fine-tuning possible for domain-specific needs.
• Popular Models:
  • msmarco-MiniLM-L-12-v3 – optimized for asymmetric search (short queries vs. long product descriptions) 3.
  • all-MiniLM-L6-v2 – fast and lightweight for general semantic tasks.
• Use Case: Ideal for large-scale product catalogs and multilingual e-commerce platforms 3 4.

3. Hybrid Search (BM25 + Semantic)

• Combine BM25 (keyword relevance) with semantic embeddings using Reciprocal Rank Fusion (RRF).
• Delivers highly relevant results by balancing literal matches and contextual meaning 1.

The post Semantic Search appeared first on Customer Experience Management.

Attribution Models and Credit

Google Ads Attribution: By default, Google Ads uses a data-driven attribution model, which gives credit to various touchpoints along the conversion path. By default ist is 30 days. It will take credit for a conversion as long as a user interacted with one of your ads within 30 days. You can defined shorter lookback window.

GA4 Attribution: GA4’s default is also data-driven, but it’s “cross-channel.” This means it considers all marketing channels (organic search, direct, social, email, etc.) in the customer journey, not just Google Ads. If a user clicks a Google ad, then later comes back to your site via organic search and converts, GA4’s data-driven model will distribute credit to both channels, while Google Ads will likely claim all or most of the conversion value.

Time-Based Reporting Google Ads Conversion and GA4 Revenue Difference

The way each platform logs a conversion can create a significant reporting gap, especially when analyzing recent data.

• Google Ads: Attributes a conversion to the date of the ad click or impression.
• GA4: Attributes a conversion to the date of the actual transaction or conversion event.

For example, if a user clicks a Google ad on September 20th but makes a purchase on October 5th, Google Ads will report the conversion value in September, while GA4 will report it in October. This “conversion lag” can cause large differences when comparing monthly or weekly reports. Google Ads Conversion and GA4 Revenue Difference.

Conversion Counting and Definitions

How you set up your conversion counting and conversion values can lead to different numbers. Especially important is conversion values for e-commerce purchase.

Conversion Counting: In Google Ads, you can choose to count a conversion “once” or “every” time it happens. For example, if a user submits a form multiple times, you might want to count it only once. However, for a purchase, you’d want to count every transaction. If these settings are not aligned between Google Ads and GA4, your numbers will not match.

Conversion Actions: If you have different conversion actions set up in Google Ads and GA4, or if they are not correctly linked, you will see a discrepancy. For example, if you track a “purchase” in GA4 but have a different conversion action for “leads” in Google Ads, the numbers will naturally be different.

Technical and User-Based Factors

These are often smaller but can add up to a substantial difference.

• Ad Blockers and User Consent: Some ad blockers and privacy settings can prevent GA4’s tracking code from firing, meaning a session and conversion might not be recorded in GA4. However, the Google Ads conversion tag is often less affected, so Google Ads may still report the conversion. Similarly, if a user opts out of tracking via a consent banner, GA4 may not receive the data.
• Quick Exits: A user may click a Google ad, be charged for the click, and then hit the back button before the GA4 tracking tag has a chance to load. Google Ads will count the click, but GA4 won’t record a session, leading to a discrepancy between clicks and sessions, and ultimately, conversions.
• Cross-Device Conversions: Google Ads uses modeled conversions to account for users who start their journey on one device and finish it on another. This can lead to a higher conversion count in Google Ads than in GA4, especially if Google Signals is not enabled in GA4.
• View-Through Conversions: Google Ads counts “view-through conversions,” which are conversions that happen after a user sees a display ad but doesn’t click on it. GA4 does not track these by default, which can cause Google Ads to report more conversions and higher conversion value.

What to Do About It

Align Your Attribution Models: Consider using the same attribution model in both platforms for a more direct comparison. While Google Ads defaults to data-driven, you can use the “Model Comparison” tool in GA4 to see how your data would look under a different model, like “Last Click.”

The Model Comparison report, also referred to as the Attribution models report, in Google Analytics 4 (GA4) is a tool that allows you to compare how different attribution models distribute credit for conversions. It helps you understand how various marketing channels, like paid search, social media, and organic search, contribute to a user’s conversion path. This is a crucial report for understanding the true value of your marketing efforts beyond just the last touchpoint.

The report lets you select a conversion event and then view how its value would be allocated to different channels under two different attribution models. You can then see a percentage change, which highlights which channels are being over or undervalued depending on the model you use.

Key Attribution Models to Compare

GA4 offers a few attribution models to choose from, with Data-driven being the default. Comparing this with other models is where the tool’s value really shines.

• Data-driven: This model uses machine learning to analyze all the touchpoints in a user’s journey, including both converting and non-converting paths. It then assigns credit based on the actual impact of each touchpoint. It’s considered the most accurate and sophisticated model.
• Paid and organic last click: This is a rule-based model that gives 100% of the conversion credit to the last channel a user clicked on before converting, ignoring any direct traffic.
• Google paid channels last click: This model gives 100% of the credit to the last Google Ads click before a conversion. If there was no Google Ads click, it defaults to the “Paid and organic last click” model.

Why is this important?

Understanding how different attribution models impact your data is critical for making smart business decisions. For instance, a “last click” model might make your paid search campaigns look like they’re driving all your conversions, leading you to undervalue channels like social media or display ads that introduce users to your brand much earlier in their journey. By using the Model Comparison report, you can see how a channel’s value changes when you shift from a last-click to a data-driven model, which can lead to better budgeting and optimization.

The video below offers a tutorial on how to use the Model Comparison Tool to analyze your traffic channels and conversion data in Google Analytics. How to Use the Model Comparison Tool in Google Analytics to Compare Your Traffic Channels

Check Your Conversion Settings: Ensure that your conversion actions are correctly set up and linked, and that the counting method is consistent across both platforms.

Check Your Date Ranges: When comparing, make sure you’re using a long enough date range (e.g., a full month or longer) to account for any conversion lag.

Enable Auto-Tagging: Make sure auto-tagging is enabled in your Google Ads account so that GA4 can properly attribute traffic back to your campaigns.

Don’t Panic: It’s normal to have some level of discrepancy. The key is to understand why the differences exist and use both platforms for what they’re best at: Google Ads for optimizing your paid campaigns and GA4 for understanding the full, multi-channel customer journey.

Read more here

The post Google Ads Conversion and GA4 Revenue Difference appeared first on Customer Experience Management.