SQL is the backbone of data-driven decision-making, and mastering it can dramatically improve how you interact with databases. Whether you’re a seasoned developer or just starting out, writing efficient, readable, and scalable SQL queries is a skill that pays off daily. In this post, I’ve compiled ten essential tips that will help you write smarter SQL—tips that I’ve learned, refined, and shared over time. These aren’t just theoretical best practices; they’re practical techniques that can make your queries faster, your code cleaner, and your debugging easier.

You can try Microsoft SQL Server from here: https://www.microsoft.com/pl-pl/sql-server/sql-server-downloads. And developer edition is here

From avoiding SELECT * to choosing the right join types, each tip is designed to help you think critically about how your queries perform and how they scale. You’ll also learn how to use indexes effectively, filter data early, and make smart choices between EXISTS and IN. Each section includes a short summary and a link to a full post where you can dive deeper into the topic. Whether you’re optimizing a legacy system or building something new, these tips will help you get the most out of SQL—and avoid common pitfalls that slow down your work.

Select Only What You Need, SQL Tips no 1

Avoid SELECT * and specify only the columns you need. This reduces data transfer, memory usage, and improves query speed. For example, instead of pulling all employee data, just select employee_id, first_name, and last_name. Read more

Optimize Join Conditions for Performance

Avoid non-SARGable joins that prevent index usage. Instead of applying functions to columns in join conditions, restructure the logic to preserve index efficiency. This dramatically improves query speed. Read the full guide

Use the PIVOT Operator for Better Reporting

Transform row-based data into columnar format using PIVOT. This is ideal for cross-tab reports and trend analysis, especially when comparing metrics across time or categories. Explore the PIVOT tutorial

Master Recursive CTEs for Hierarchical Data

Recursive Common Table Expressions (CTEs) allow you to elegantly query hierarchical or tree-structured data. They’re powerful for tasks like organizational charts or category trees. Learn about recursive CTEs

Set the First Day of the Week with DATEFIRST

Use SET DATEFIRST to control how SQL Server interprets weekday numbers. This is crucial for accurate time-based reporting and week-based aggregations. See how to use DATEFIRST

Update Multiple Tables with Conditions

Learn how to structure multi-table updates using joins and conditional logic. This technique is essential for synchronizing data across related tables. Read the multi-table update example

Filter Early with WHERE Clauses

Apply filters as early as possible to reduce the number of rows processed in joins and aggregations. Optimize your filtering

Use UNION ALL Instead of UNION

UNION ALL is faster than UNION because it skips duplicate elimination. Use it when duplicates aren’t a concern. Performance tip explained

Avoid Functions on Indexed Columns

Using functions like LOWER() or DATEADD() on indexed columns disables index usage. Rewrite conditions to preserve index paths. Join optimization example

Explore SQL for Business Analytics

Advanced SQL techniques like statistical analysis, predictive modeling, and time series forecasting are essential for business analysts. Learn how to combine SQL with Python for deeper insights. Check out the full course

5 additional SQL tips

Use CTEs for Readability and Reuse

Common Table Expressions (CTEs) make complex queries easier to read and maintain. They allow you to define temporary result sets that can be referenced multiple times. SQL Tips

WITH recentorders AS (
  SELECT customerid, orderdate
  FROM orders
  WHERE orderdate > CURRENTDATE - INTERVAL '30 days'
)
SELECT customerid, COUNT(*) AS ordercount
FROM recentorders
GROUP BY customer_id;

Avoid Functions on Indexed Columns

Using functions on indexed columns disables index usage, slowing down queries. Instead, transform the value before comparison. SQL Tips

-- Avoid
SELECT FROM users WHERE LOWER(email) = 'test@example.com';
-- Better
SELECT FROM users WHERE email = 'test@example.com';

Use CASE for Conditional Logic

CASE lets you embed conditional logic directly in your queries, useful for categorizing or transforming data.

SELECT name,
  CASE
    WHEN score >= 90 THEN 'Excellent'
    WHEN score >= 75 THEN 'Good'
    ELSE 'Needs Improvement'
  END AS performance
FROM students;

Optimize Aggregations with GROUP BY

When aggregating data, ensure you’re grouping only necessary columns to avoid performance hits and incorrect results.

SELECT department, AVG(salary) AS avg_salary
FROM employees
GROUP BY department;

Use Parameterized Queries to Prevent SQL Injection

Always use parameterized queries in application code to protect against SQL injection.

-- Example in Python with psycopg2
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))

Remember – tips summary

Use Joins Efficiently

Choose the right join type—INNER JOIN for matched rows, and avoid CROSS JOIN unless necessary. Efficient joins reduce unnecessary data processing and improve clarity.

Filter Data Early

Apply filters as soon as possible in your query using WHERE clauses. This minimizes the number of rows processed in joins and aggregations, leading to faster execution. SQL Tips no 3

Use Indexes Wisely

Indexes speed up data retrieval, especially in WHERE, JOIN, and ORDER BY clauses. But don’t over-index—too many can slow down INSERT, UPDATE, and DELETE operations.

Avoid Subqueries in WHERE Clauses

Correlated subqueries can be slow. Replace them with joins when possible to improve performance and readability. SQL Tips no 4

Use UNION ALL Instead of UNION

UNION removes duplicates, which is costly. If duplicates aren’t a concern, use UNION ALL for faster results.

Limit Your Results

Use LIMIT or TOP to restrict the number of rows returned. This is especially useful for pagination or sampling large datasets.

Be Cautious with LIKE and Functions

Avoid leading wildcards in LIKE and functions in WHERE clauses—they prevent index usage. Instead, use indexed columns and consistent casing.

Use EXISTS Instead of IN

EXISTS is often faster than IN because it stops scanning once a match is found. Use it for subqueries checking row existence.

Use Appropriate Data Types

Choosing the right data type—like TINYINT over INT or CHAR over VARCHAR—can save space and improve performance.

SQL Server Management Studio

SSMS as a Comprehensive SQL Environment
SQL Server Management Studio (https://learn.microsoft.com/en-us/ssms/) is a powerful, integrated environment for managing SQL Server infrastructure. It provides tools for writing, executing, and debugging SQL queries, as well as managing databases, tables, views, and stored procedures. SSMS supports both on-premises and cloud-based SQL Server instances, making it versatile for hybrid environments. Its intuitive interface includes Object Explorer for navigating server components and Query Editor for crafting and testing SQL scripts. Whether you’re a database administrator or developer, SSMS offers a unified workspace that streamlines daily tasks and enhances productivity through built-in templates, syntax highlighting, and error diagnostics.

Security, Performance, and Monitoring Tools
SSMS includes robust features for security management, such as configuring roles, permissions, and auditing access. It also provides performance tuning tools like the Database Engine Tuning Advisor and graphical execution plans to help identify bottlenecks. With Activity Monitor, users can track real-time server performance, view active sessions, and analyze resource usage. These tools empower teams to maintain optimal database health and ensure compliance with organizational policies. SSMS also integrates with SQL Server Agent for scheduling jobs and alerts, making it a central hub for automation and proactive monitoring across enterprise environments.

Integration, Extensibility, and Collaboration
SSMS supports integration with source control systems like Git, enabling versioning and collaborative development. It allows exporting and importing data via wizards, scripting database objects, and generating reports for documentation. Users can extend SSMS functionality through add-ins and connect to Azure services for cloud-based analytics and storage. Its support for multiple query windows and tabbed editing enhances multitasking, while customizable keyboard shortcuts and themes improve user experience. SSMS continues to evolve with regular updates, ensuring compatibility with the latest SQL Server features and providing a stable platform for modern data operations.

The post 15 Essential SQL Tips You Can’t Live Without appeared first on Customer Experience Management.

Frequency Analysis for Profiling

This is the most direct approach to identify the most and least selected lectures by your L6 students.

1. Filter Data: Isolate the rows (students) who selected L6.
2. Calculate Frequencies: For this subset of L6 students, count how many of them selected each of the other available lectures (L1, L2, L3, etc.).
   • Most Selected: The lectures with the highest counts.
   • Least Selected: The lectures with the lowest counts (or those not selected at all).

Association Rule Mining – Co-occurrence

This sophisticated approach, often used in market basket analysis, can determine which lectures are most frequently chosen together with L6.

• Support: The proportion of L6 students who also selected a specific lecture (e.g., L3).
• Confidence: The likelihood that a student selected L6 given that they selected another lecture (e.g., L3 → L6), or vice-versa.
• Lift: A measure of how much more likely a student is to select L3 if they also selected L6, compared to the overall likelihood of selecting L3. A Lift >1 suggests a positive association (students who take one tend to take the other).

Correlation Analysis – the Strength and Direction of Relation – application in Profiling

This method quantifies the relationship between selecting L6 and selecting any other lecture (Lx). Since the selection data is binary (0 for not selected, 1 for selected), you would use a correlation measure suitable for binary variables. Profiling.

• Phi Coefficient (ϕ): This is a measure of association for two binary variables. It ranges from −1 to +1.
  • Strong Positive Correlation (ϕ≈+1): Students who select L6 are highly likely to also select Lx. This suggests the lectures are perhaps complementary or targeted at the same student group.
  • Strong Negative Correlation (ϕ≈−1): Students who select L6 are highly likely to not select Lx. This suggests the lectures might be alternatives, require conflicting time slots, or appeal to entirely different student interests.
  • Weak/No Correlation (ϕ≈0): Selection of L6 has little to no impact on the selection of Lx.

Dimensionality Reduction – Clustering

For a very large number of lectures, you could use methods like Principal Component Analysis (PCA) or clustering algorithms to group similar students or lectures together. This can identify underlying student profiles (e.g., “The Data Science Crowd” or “The Humanities Enthusiasts”) that include L6 as part of their typical selection.

In this table there are students (in rows) selection of lectures (in column). My lecture is L6. I would like to know a profile of my students. so which lectures they selected the most and which the least. How strong the relations are (positive and negative if they omit some lectures.

Profile of L6 Students: Most and Least Selected Lectures

This is a Frequency Analysis of the lectures selected by the 11 students who chose L6. The percentages are based on the total number of L6 students (15).

Most Selected Lectures – The “Typical Package”: Your L6 students most frequently select L4 (67%), L3 (53%), and L5 (53%). These three form the core lecture package with L6. Least Selected Lecture: L12 is the least popular choice, selected by only 3 out of 15 students (20%).

Strength of Relation: Correlation Analysis

The Phi Coefficient quantifies the strength and direction of the relationship between choosing L6 and choosing any other lecture, using all 25 students in the dataset.

Strongest Positive Relations (Complementary): L4 (ϕ=0.263) and L5 (ϕ=0.230) show the strongest positive correlation with L6. This suggests that students interested in L6 are often those who also select L4 and L5.

Strongest Negative Relation (Alternative/Avoided): L12 (ϕ=−0.218) shows the only notable negative correlation. This confirms the frequency finding, suggesting L12 may be an alternative path or have a conflicting time/prerequisite with L6.

Weak/No Relation: Lectures like L3 and L7 have a high selection frequency but a very weak (L3) or zero (L7) correlation. This indicates that while many L6 students do take these, they are likely popular lectures chosen by many students across the board, and the choice of L6 is not a significant predictor of their selection.

Association Rule Mining

By analyzing the entire student population, we can discover general student curriculum patterns that exist beyond your specific L6 cohort. I used Association Rule Mining metrics (Support, Confidence, and Lift) to find lecture pairs that are frequently selected together.

• Support: The percentage of all 25 students who selected both lectures.
• Lift: A measure of how much the selection of one lecture increases the chance of selecting the other. A Lift>1.2 indicates a strong, meaningful positive association.

Here are the top co-selected lecture groups (pairs) among the entire student population, filtered for those selected by at least 16% of students and showing a strong positive association (Lift>1.2):

Python

import pandas as pd
from io import StringIO
import itertools

# Read the data
df = pd.read_csv(StringIO(csv_data), sep=';')
N_students = len(df)

lecture_cols = [col for col in df.columns if col.startswith('L')]
lecture_df = df[lecture_cols].fillna(0).astype(int)

# --- Association Rule Mining (Pairs) ---
# 1. Calculate Support for individual lectures
support_single = lecture_df.sum() / N_students

# 2. Calculate Support and Lift for all pairs
association_rules = []
for l1, l2 in itertools.combinations(lecture_cols, 2):
    # Calculate Support for the pair: count students who selected both
    co_selection_count = (lecture_df[l1] * lecture_df[l2]).sum()
    support_pair = co_selection_count / N_students

    # Calculate Lift: (Support(L1 and L2)) / (Support(L1) * Support(L2))
    # Handle division by zero if single support is 0, though unlikely here
    if support_single[l1] > 0 and support_single[l2] > 0:
        lift = support_pair / (support_single[l1] * support_single[l2])
    else:
        lift = 0

    # Calculate Confidence (L1 -> L2)
    confidence_l1_to_l2 = support_pair / support_single[l1] if support_single[l1] > 0 else 0

    association_rules.append({
        'Antecedent': l1,
        'Consequent': l2,
        'Support': support_pair,
        'Confidence (L1 -> L2)': confidence_l1_to_l2,
        'Lift': lift
    })

# Convert to DataFrame
rules_df = pd.DataFrame(association_rules)

# Filter for meaningful associations:
# 1. Minimum Support: Selected by at least 4 students (4/25 = 0.16)
# 2. Lift > 1.2: A strong positive relationship
min_support = 4 / N_students  # 0.16

# Filter and sort the results by Lift
top_associations = rules_df[
    (rules_df['Support'] >= min_support) &
    (rules_df['Lift'] > 1.2)
].sort_values(by='Lift', ascending=False).reset_index(drop=True)

# Add the reverse rules (L2 -> L1) to the table where Lift is high.
# Since Lift is symmetrical, only one direction needs to be calculated, but Confidence is not.

# Helper function to get Confidence (L2 -> L1) for presentation
def get_confidence_l2_to_l1(row):
    l1 = row['Antecedent']
    l2 = row['Consequent']
    support_pair = row['Support']
    return support_pair / support_single[l2] if support_single[l2] > 0 else 0

top_associations['Confidence (L2 -> L1)'] = top_associations.apply(get_confidence_l2_to_l1, axis=1)

# Reorder columns for presentation
top_associations = top_associations[['Antecedent', 'Consequent', 'Support', 'Confidence (L1 -> L2)', 'Confidence (L2 -> L1)', 'Lift']]

print("Top Co-Selected Lecture Groups (Pairs):")
print(top_associations.to_markdown(index=False, floatfmt=".3f"))

import pandas as pd
from io import StringIO
import itertools

# Read the data
df = pd.read_csv(StringIO(csv_data), sep=';')
N_students = len(df)

lecture_cols = [col for col in df.columns if col.startswith('L')]
lecture_df = df[lecture_cols].fillna(0).astype(int)

# --- Association Rule Mining (Pairs) ---
# 1. Calculate Support for individual lectures
support_single = lecture_df.sum() / N_students

# 2. Calculate Support and Lift for all pairs
association_rules = []
for l1, l2 in itertools.combinations(lecture_cols, 2):
    # Calculate Support for the pair: count students who selected both
    co_selection_count = (lecture_df[l1] * lecture_df[l2]).sum()
    support_pair = co_selection_count / N_students

    # Calculate Lift: (Support(L1 and L2)) / (Support(L1) * Support(L2))
    # Handle division by zero if single support is 0, though unlikely here
    if support_single[l1] > 0 and support_single[l2] > 0:
        lift = support_pair / (support_single[l1] * support_single[l2])
    else:
        lift = 0

    # Calculate Confidence (L1 -> L2)
    confidence_l1_to_l2 = support_pair / support_single[l1] if support_single[l1] > 0 else 0

    association_rules.append({
        'Antecedent': l1,
        'Consequent': l2,
        'Support': support_pair,
        'Confidence (L1 -> L2)': confidence_l1_to_l2,
        'Lift': lift
    })

# Convert to DataFrame
rules_df = pd.DataFrame(association_rules)

# Filter for meaningful associations:
# 1. Minimum Support: Selected by at least 4 students (4/25 = 0.16)
# 2. Lift > 1.2: A strong positive relationship
min_support = 4 / N_students  # 0.16

# Filter and sort the results by Lift
top_associations = rules_df[
    (rules_df['Support'] >= min_support) &
    (rules_df['Lift'] > 1.2)
].sort_values(by='Lift', ascending=False).reset_index(drop=True)

# Add the reverse rules (L2 -> L1) to the table where Lift is high.
# Since Lift is symmetrical, only one direction needs to be calculated, but Confidence is not.

# Helper function to get Confidence (L2 -> L1) for presentation
def get_confidence_l2_to_l1(row):
    l1 = row['Antecedent']
    l2 = row['Consequent']
    support_pair = row['Support']
    return support_pair / support_single[l2] if support_single[l2] > 0 else 0

top_associations['Confidence (L2 -> L1)'] = top_associations.apply(get_confidence_l2_to_l1, axis=1)

# Reorder columns for presentation
top_associations = top_associations[['Antecedent', 'Consequent', 'Support', 'Confidence (L1 -> L2)', 'Confidence (L2 -> L1)', 'Lift']]

print("Top Co-Selected Lecture Groups (Pairs):")
print(top_associations.to_markdown(index=False, floatfmt=".3f"))

Interpretation of Lecture Groups

The Lift values indicate the strength of the relationships:

The Strongest Cohort (The 2.2+ Lift Groups)

These are the three strongest, non-obvious combinations. A Lift≈2.2 means that students who take one lecture are more than twice as likely to take the associated lecture compared to the general student population.

• L9 ↔ L12 ↔ L8: These three lectures form a tight cluster. Specifically, L8 → L9 and L9 → L12 have extremely high association scores. This suggests a dedicated academic track or a highly correlated set of topics.

High Confidence Groups (The 1.5+ Lift Groups)

These are groups where the selection of one lecture is a very strong predictor for the other:

• L2 → L4 (87.5% Confidence): If a student selects L2, there is a very high, 87.5% chance they also select L4. This suggests L2 might be a prerequisite, a foundational course, or a direct complement to L4.
• L2 → L3 (75.0% Confidence): Similarly, L2 and L3 are frequently chosen together, indicating a strong connection.
• L1 ↔ L2: These two lectures are highly associated, suggesting they are often taken in tandem.

Does this mean that I should go to lecturers L5 and L2 and ask them for promotion of my L6, because it is most likely they find satisfactory experience on my lecture?

Data-Driven Strategy for L6 Promotion

1. Prioritize L5 (High Co-Selection & High Confidence)

You should absolutely focus on the L5 lecturer.

• L5 → L6 Confidence: 72.7%
  • This means nearly 3 out of every 4 students who take L5 also choose your L6 lecture.
  • Action: The L5 lecturer is teaching the same target audience as you. You could ask them to mention L6 as a natural follow-up or complementary course to their students.

2. Prioritize L2 (High Predictive Power)

The relationship with L2 is even more predictive of a student landing in your lecture.

• L2 → L6 Confidence: 75.0%
  • This means 3 out of every 4 students who take L2 end up in L6.
  • Action: The L2 lecturer is essentially teaching a foundational course for a majority of your class. Ask them to promote L6 as the direct next step or most relevant application course for their content.

3. The Importance of Satisfaction (What the Data Doesn’t Say)

The data confirms a strong selection link, but not a satisfaction link.

• Selection: L5 and L2 students are your target market.
• Satisfaction: To confirm they find your lecture satisfactory, you’d need student feedback/evaluation data. A student who disliked L6 might still be highly likely to take it if it’s a required course for a specific program, for example.

The strongest rationale for promotion is simply the high overlap: you are addressing a student cohort that already has a demonstrated interest pattern (L5/L2 → L6).

Secondary Focus: L4 and L3

While L4 and L3 have high total co-selection with L6 (L4: 66.7%, L3: 53.3%), their Lift and correlation scores are weaker. This means they are generally popular lectures, and while many of your students take them, the choice of L4 or L3 does not strongly predict the choice of L6.

Focusing your limited promotional time on the lecturers for L5 and L2 will yield the highest return.

Country and University as moderating factors

The factors of Country and University do show clear patterns in lecture selection among your L6 students, suggesting that existing relationships or shared academic paths likely influence their choices.

Here is the analysis of the moderating factors, based on the 15 students in your L6 lecture:

Analyzing the two largest university groups shows even sharper differences, which is expected as they are likely organized groups of students who know each other.

Theoretical Background for Moderating Factors

The Influence of Country: Cultural and Institutional Homophily

The tendency for students from the same country (e.g., Spain or Morocco) to share similar lecture profiles can be explained by Homophily and Institutional Alignment.

The Influence of University/Peer Group: Social Network Effects

The even stronger, more granular influence of the specific university groups (like the unique L7 selection by U. Malaga students) is supported by Social Influence Theory and Bounded Rationality.

By applying these concepts, you can elevate your analysis from mere correlation to a discussion of student decision-making under conditions of uncertainty and social constraint.

The lecture title informational role

Assuming that students red only the lecture titles can you analyze the whole population and discover which lectures students select as a group (l1 & l2 … ect.) based on title informational role.

This is an excellent way to use the data to understand the thematic logic that guides students’ decisions, rather than just the frequency. By analyzing the corpus of words in the high-Lift lecture titles, we can discover the “driving wording” that creates strong, predictable co-selection patterns.

The primary driving themes are Economics, Management, and Business, but the strongest associations occur when these words are combined with qualifying terms like International, Applied, or Process.

Analysis of Driving Wording in Lecture Titles

Based on the highest Lift scores (strongest association patterns), the lecture groups cluster into four distinct thematic tracks driven by specific keywords:

1. Driving Theme: International & Political Economy 🌍

This is the strongest thematic driver in the entire dataset, creating three of the top four co-selection groups.

Driving Wording: International, Economy, Political.

2. Driving Theme: Applied Economic Analysis

This theme links foundational economic knowledge with quantitative skills and real-world application.

Driving Wording: Applied, Analysis, Economics, Decisions.

3. Driving Theme: Business Management & Strategy

This group is driven by a focus on business processes and the organizational changes brought by technology.

Driving Wording: Management, Process, Digital, Innovation.

Summary of Driving Wording

The strongest selection drivers are not single words but rather qualified thematic phrases that define an academic specialization:

Modern Business Specialization: Driven by Management combined with Process or Digital/Innovation (L5, L6, L7, L11).

Macro/Global Specialization: Driven by International and Political Economy (L9, L12, L8).

Quantitative Specialization: Driven by Applied Economics and Business Analytics (L2, L3, L1).

About wording analysis you can read here: Organizational aspiration for social impact

What recommendation to lecture L6 in wording. How can I modify or extend title of L6 to experiment for more students in next period.

The goal is to maximize enrollment by aligning the L6 title with the thematic drivers of the most predictive lectures (L5, L2, L4). Your current title is L6: Customer Experience Management.

Here are the recommendations for wording modifications, structured by the target student profile you want to attract:

Recommended Title Modifications for L6

Strategy 1: Attract L5 Students (Strongest Existing Link)

The L5 title is Business process management. These students seek internal efficiency as a foundation for external success.

Strategy 2: Attract L2/L4 Students (Applied & Analytical)

The L2 title is Applied Economics Real-World Challenges and Solutions. L4 is Business plan. These students are practical and analytical.

Strategy 3: Attract L7/L11 Students (Future/Digital Focus)

The L7/L11 titles feature Innovation and Digital Economy.

Final Recommendation: The Best Title to Test

The most balanced and powerful title that leverages multiple co-selection themes is: Customer Experience Management: Data, Process, and Digital Strategy

This phrase:

• Includes Management (L5, L6, L11 theme).
• Includes Process (L5 link).
• Includes Data (L2/L3 Analytics link).
• Includes Digital Strategy (L7/L11 Innovation link).

Thank you for reading.

The post Student Profiling for your lecture appeared first on Customer Experience Management.

The symmetric data analysis (e.g., correlation and multiple regression analysis) …

The asymmetric data analysis (i.e., individual case outcome forecasts) …

Based on: Fuzzy-set Qualitative Comparative Analysis (fsQCA): Guidelines for research practice in Information Systems and marketing (Ilias O. Pappas, Arch G. Woodside)

Qualitative inductive reasoning with data being analyzed “by case’’ and not “by variable’’, is combined with quantitative empirical testing, as sufficient and necessary conditions identify outcomes through statistical methods. In most cases, QCA are useful in quantitative studies, as it allows the researcher to get a deep view of their data through a quantitative analysis. That has also several characteristics of qualitative analysis.

Case studies focus on describing, explaining, and forecasting, single and combinatorial conditional antecedents on outcomes while variable studies focus on the similarities of variances of two or more variables. A “condition” is a point or interval range of antecedent or outcome; a “variable” characteristic varies.

Here are few examples of conditions versus variables: “Male” is a condition; “gender” is a variable. “Swedish” is a condition; “nationality” is a variable. “Expert” is a condition; “expertise” is a variable.

The goal of QCA is to explain causality in complex real life phenomena. QCA goes through “multiple-conjunctural causation, which refers to “nonlinear, nonadditive, non-probabilistic conception that rejects any form of permanent causality. That stresses different paths which can lead to the same outcome. QCA investigate complex combinations of conditions and diversity. QCA uses Boolean algebra and Boolean minimization algorithms to capture patterns of multiple-conjunctural causation and to simplify complex data structures.

Types qualitative comparative analysis (QCA)

CsQCA and mvQCA

CsQCA is the first variation of QCA. It is a tool created to deal with complex sets of binary data. The use of Boolean algebra means that QCA has as input binary data (0 or 1). That make QCA uses logical operations for the procedure. Thus it is very important to dichotomize the use of variables in a useful and meaningful manner.

mvQCA, treats variables as multi-valued instead of dichotomous. MvQCA retains the idea of performing a synthesis of the dataset and cases with the same value on the outcome variable. They are explained by a solution, which contains combinations of variables that explain a number of cases with the outcome.

FsQCA addresses an important limitation of csQCA, the fact that variables are binary, thus restricting the analysis as it cannot fully capture the complexity in cases that naturally vary by level or degree. This restriction of csQCA is likely an important reason that QCA has not been widely adopted in multiple contexts, including IS and marketing research. FsQCA extends csQCA by integrating fuzzy-sets and fuzzy-logic principles with QCA. The variables can get all the values within the range of 0–1. FsQCA is able to overcome several limitations of both csQCA and mvQCA, and has received increased attention recently. FsQCA applies together with complexity theory, it provides the opportunity to gain deeper and richer insight into data.

FsQCA and cluster analysis

Case-based techniques, such as fsQCA and cluster analysis, have been employed as a way of moving beyond variance-based methods. These two techniques have similarities as they both employ multidimensional spaces and often people ask how fsQCA differs from cluster analysis and why do we need it. A main difference between the two methods is the kind of research questions they are able to address.

Specifically, cluster analysis answers questions such as which cases are more similar to each other, while fsQCA can identify the different configurations that constitute sufficient and/or necessary conditions for the outcome of interest. Depending on the focus of the study the researcher should choose the most appropriate method. Their differences stem from the fact that “QCA addresses the positioning of cases in [multidimensional] spaces via set theoretic operations while cluster analysis relies on geometric distance measures and concepts of variance minimization” . To this end, prior studies compare fsQCA with cluster analysis and show how fsQCA can handle causal complexity with fine-grained level data, or how it can identify more solutions compared to cluster analysis. A discussion exists in the literature regarding QCA and cluster analysis, and both approaches have differences making them suitable for different types of studies.

read example here: Hierarchical Agglomerative Clustering for Product Grouping

Adoption of fsQCA in relevant studies

Configurational approaches are becoming more popular over the past few years in different areas, with fsQCA playing a large part in this as most studies will prefer fuzzy-set over crisp-set and multi-value QCA (Thiem & Dusa, 2013). In detail, fsQCA has been employed in information systems (Fedorowicz, Sawyer, & Tomasino, 2018; Liu et al., 2017), online business and marketing (Pappas et al., 2016; Pappas, 2018; Woodside, 2017), consumer psychology (Schmitt, Grawe, & Woodside, 2017), strategy and organizational research (Fiss, 2011; Greckhamer et al., 2018), education (Pappas, Giannakos et al., 2017; Plewa, Ho, Conduit, & Karpen, 2016), data science (Vatrapu, Mukkamala, Hussain, & Flesch, 2016) and learning analytics (Papamitsiou et al., 2018; Sergis, Sampson, & Giannakos, 2018). This tutorial aims to increase the adoption of fsQCA in IS and marketing studies following the call for more empirical work in the area (El Sawy, Malhotra, Park, & Pavlou, 2010; Fiss, 2011; Woodside, 2014, 2017).

Adoption of fsQCA in relevant studies

FsQCA is useful for both inductive and deductive reasoning for theory building, elaboration, and testing. This analysis allows to identify specific cases in the sample. With this knowledge, the researcher can go back to the cases and use contextual information, not including in the analysis, to further explain and discuss the findings.

A typical variance-based analysis would identify a single best solution, thus limiting the results. FsQCA studies can compare the findings between different data analysis techniques to describe how different stories are hidden in the same dataset. It is recommended to combine fsQCA with other data analysis techniques if possible.

How to use fsQCA in a typical e-commerce study

Sampling

The study examined cognitive and affective perceptions as antecedents of online shopping behavior in personalized e-commerce environments. We used a typical a snowball sampling methodology to recruit participants and controlled for respondents’ previous experience with both online shopping and personalized services. Eventually, the sample comprises 582 individuals with experience in online shopping and personalized services. We collected data through a questionnaire built with measures that have been adopted from the literature. Appendix A (as presented in the original study) lists construct definitions, the questionnaire items used to measure each construct, along with descriptive statistics and loadings.

Evaluate constructs for reliability and validity

Typical with similar quantitative studies, first we evaluate constructs for reliability and validity. This is a step that should be always performed when it is appropriate, and it is not directly related with the fsQCA analysis as it depends on the type of variables that are used in the study. Construct reliability and validify, as the name implies, refer to the construct itself and not to the method of analysis used to examine relations between constructs.

Reliability testing, based on the Cronbach alpha indicator, showed acceptable indices of internal consistency since all constructs exceed the cut-off threshold of 0.70. The AVE for all constructs ranged between 0.55 and 0.84, all correlations were lower than 0.80, and square root AVEs for all constructs were larger than their correlations. The findings in detail for the confirmatory analysis may be found in the original paper.

Contrarian case analysis

Contrarian case analysis is performed outside fsQCA, but we present it here because it can serve as an easy and quick way to examine how many cases in our sample are not explained by main effects, and thus they would not be included in the outcome of a typical variance-based approach, e.g., correlation or regression analysis.

Data Calibration

In fsQCA, different from traditional methods, instead of working with probabilities data are transformed from ordinal or interval scales into degrees of membership in the target set, which shows if and how much a case belongs into a specific set. “In essence, a fuzzy membership score attaches a truth value, not a probability, to a statement”.

For example, the variable intention to purchase can be coded as “high intention to purchase”, and we will be looking for the presence or absence of the condition high intention to purchase (“intention to purchase” is the variable; “high intention to purchase” is a condition). Similarly, we code the rest of the variables.

The method computes the presence of a condition or its opposite (i.e., negation). The negation of a condition is referred in the literature as the absence of a condition, and the two terms have been used interchangeably based on how the absence is computed. The term absence has been also used to describe when the condition is irrelevant in a configuration. It is similar to the “do not care” term that is also often used in the literature.

This distinction is not often addressed or clarified, thus we suggest researchers to clearly define these terms in future works to avoid misunderstandings.

Transform data into fuzzy-sets

In fsQCA we need to calibrate our variables to form fuzzy sets with their values ranging from 0 to 1. Consider a fuzzy set as a group, then the values from 0 to 1 define if and at what amount a case belongs to this group. The fact that all values range from 0 to 1 means that a case with a fuzzy membership score of 1 is a full member of a fuzzy set (fully in the set), and a case with a membership score of 0 is a full non-member of the set (fully out of the set). A membership score of 0.5 is exactly in the middle, thus a case would be both a member of the fuzzy set and a non-member, and is therefore a member of what is known as the intermediate set. The intermediate-set point is the value where there is maximum ambiguity as to whether a case is more in or more out of the target set.

Data calibration may be either direct or indirect. In the direct calibration the researcher needs to choose exactly three qualitative breakpoints, which define the level of membership in the fuzzy set for each case (fully in, intermediate, fully out). In the indirect method, the measurements need to be rescaled based on qualitative assessments. The researcher may choose to calibrate a measure differently, depending on what one is investigating. Either method may be chosen, depending on researcher’s substantive knowledge of both data and underlying theory. The direct method is recommended and is more common, in which the researcher sets three values corresponding to full-set membership, full-set non-membership, and intermediate-set membership. This can lead to more rigorous studies which are easier to be replicated and validated, since it is clearer on how the thresholds have been chosen.

The percentiles allow the calibration of any measure regardless of its original values. In detail, we can compute the 95 %, 50 %, and 5 % of our measures and use these values as the three thresholds in fsQCA software.

Especially in the case of the widely used seven-point Likert scales (1=Not at all, 7=Very much), previous studies suggest that the values of 6, 4, and 2 can be used as thresholds. Similarly, for a five-point Likert scale the thresholds could be 4,3, and 2.

Interpreting and presenting the solutions

FsQCA software provides all three solutions every time. Complex and parsimonious solutions are computed regardless of any simplifying assumptions employed by the researcher (e.g., choosing the presence or absence/negation of a variables) while the intermediate solution depends on these assumptions. While the intermediate solution includes both core and peripheral conditions, we need an easy way to make the distinction that will help us interpret and present the solutions in a better manner.

To improve the presentation of the findings we can transform the solutions from fsQCA output into a table that is easier to read. Typically,

1. the presence of a condition is indicated with a black circle (●),
2. the absence/negation with a crossed-out circle (⊗),
3. and the “do not care” condition with a blank space.

The negation of a condition is referred in the literature also as absence, and the two terms have been used interchangeably. The distinction between core and peripheral is made by using large and small circles, respectively. The researcher needs to present the overall solution consistency and the overall solution coverage. The overall coverage describes the extent to which the outcome of interest may be explained by the configurations, and is comparable with the R-square reported on regression-based methods. In our example, the results indicate an overall solution coverage of 0.84, which suggests that a substantial proportion of the outcome is covered by the nine solutions.

All graphics and futher explanation you find here: Pappas, I. O., & Woodside, A. G. (2021). Fuzzy-set Qualitative Comparative Analysis (fsQCA): Guidelines for research practice in Information Systems and marketing. International journal of information management, 58, 102310.

The post Fuzzy-Set Qualitative Comparative Analysis appeared first on Customer Experience Management.

Independent variables in simple terms:

• Independent variables: Inputs you control or observe.
• Dependent variable: Output you want to understand or predict.

Example of independent variables in Customer Management (RFM Model):

Suppose you’re analyzing customer behavior to predict churn (whether a customer will stop buying).

• Independent variables:
  • Recency: How recently a customer made a purchase.
  • Frequency: How often they purchase.
  • Monetary: How much they spend.
• Dependent variable:
  • Churn: 1 if the customer churned, 0 if they stayed.

In this case, Recency, Frequency, and Monetary are independent variables used to predict the likelihood of churn. See also Python for business analytics – rfm analysis

Why check for independence among independent variables?

If independent variables are highly correlated with other variables, i.e., it is not truly independent, it can cause multicollinearity, which makes model coefficients unstable, reduces interpretability, and can lead to misleading conclusions.

Multicollinearity

Multicollinearity refers to a statistical phenomenon in which two or more independent variables in a regression model are highly correlated. This makes it difficult to determine the individual effect of each variable on the dependent variable because they essentially carry overlapping information.

Assessment of Multicollinearity

To assess multicollinearity, you can use following methods:

1. Correlation Matrix
   • Check pairwise correlations between independent variables.
   • High correlation (e.g., > 0.8 or < -0.8) may indicate multicollinearity.
2. Variance Inflation Factor (VIF)
   • Measures how much the variance of a regression coefficient is inflated due to multicollinearity.
   • VIF > 5 or 10 is often considered problematic.
3. Tolerance
   • Tolerance = 1 / VIF.
   • Low tolerance values (close to 0) indicate high multicollinearity.
4. Condition Index and Eigenvalues
   • Part of a more advanced diagnostic using matrix decomposition.
   • A condition index > 30 may suggest serious multicollinearity.

How to deal with multicollinearity?

• Remove one of the correlated variables.
• Combine variables (e.g., using PCA or creating an index).
• Regularization techniques like Ridge or Lasso regression.
• Centering variables (subtracting the mean) can help in some cases.

Calculation example

Assume, you have data similar to this sample.

Variable independence testing

import pandas as pd

# Load the data with the specified delimiter
df = pd.read_csv("RFM_analysis_614.csv", delimiter=",")

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Select the independent variables (RFM)
X = df[['recency', 'freq', 'monetary']]

# Add a constant for the VIF calculation (required by the statsmodels function)
X = add_constant(X)

# Create a DataFrame to hold the VIF results
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

# Exclude the constant row from the final output since it's not a true variable
vif_data = vif_data[vif_data.Variable != 'const'].reset_index(drop=True)

print(vif_data)

# Save the VIF results to a CSV file
vif_data.to_csv("vif_results.csv", index=False)

import pandas as pd

# Load the data with the specified delimiter
df = pd.read_csv("RFM_analysis_614.csv", delimiter=",")

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Select the independent variables (RFM)
X = df[['recency', 'freq', 'monetary']]

# Add a constant for the VIF calculation (required by the statsmodels function)
X = add_constant(X)

# Create a DataFrame to hold the VIF results
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

# Exclude the constant row from the final output since it's not a true variable
vif_data = vif_data[vif_data.Variable != 'const'].reset_index(drop=True)

print(vif_data)

# Save the VIF results to a CSV file
vif_data.to_csv("vif_results.csv", index=False)

Finally program prints following results

Based on the Variance Inflation Factor (VIF) calculation, the columns recency, frequency, and monetary are statistically independent of each other and not informationally overlapping.

This means that you can use all three variables together as independent predictors in a statistical model, such as a Cox Proportional Hazards (Cox PH) model, without concern for severe multicollinearity.

Variance Inflation Factor (VIF) Results

The VIF (https://en.wikipedia.org/wiki/Variance_inflation_factor) is a measure of how much the variance of an estimated regression coefficient is increased due to collinearity. A common rule of thumb is that a VIF value less than 5 or sometimes 10 indicates that the correlation between the variables is not high enough to warrant concern.

The calculated VIF values for your RFM variables are very low:

Conclusion on Independence

Since all VIF values are close to 1.0 and well below the 5.0 threshold:

• Independent Variables: You can confidently treat recency, frequency, and monetary as independent variables for your statistical analysis (e.g., in a Cox PH model).
• No Informational Overlap: The variables are providing distinct, non-redundant information to the model. For instance, knowing a customer’s frequency does not allow the model to strongly predict their recency or monetary value.

The post What are independent variables? appeared first on Customer Experience Management.

1. PyCharm

• Developer: JetBrains
• Platforms: Windows, macOS, Linux
• Features: Advanced code analysis, smart code completion, integrated debugger, Git support, virtual environment management, Django support.
• Community (free) and Professional (paid) editions.
• Website: PyCharm

2. Visual Studio Code (VS Code)

• Developer: Microsoft
• Platforms: Windows, macOS, Linux
• Features: Lightweight but powerful; excellent Python extension; integrated terminal; rich plugin ecosystem; Git support; Jupyter notebook integration.
• Website: VS Code

3. Spyder

• Developer: Scientific Python Development Environment Community
• Platforms: Windows, macOS, Linux
• Features: Focused on scientific computing and data science; variable explorer; integrated IPython console; plotting support.
• Website: Spyder

4. Thonny

• Developer: University of Tartu
• Platforms: Windows, macOS, Linux
• Features: Beginner-friendly; simple UI; built-in debugger; good for learning and education.
• Website: Thonny

5. Wing IDE

• Developer: Wingware
• Platforms: Windows, macOS, Linux
• Features: Powerful debugger; code intelligence; remote development support.
• Website: Wing IDE

6. Eric

• Developer: Detlev Offenbach
• Platforms: Windows, macOS, Linux
• Features: Full-featured Python and Ruby IDE; integrated debugger; plugin support.
• Website: Eric Python IDE

7. IDLE

• Developer: Python Software Foundation (bundled with Python)
• Platforms: Windows, macOS, Linux
• Features: Basic, lightweight, good for quick scripts and learning.
• Website: IDLE Documentation

Summary:
For professional development, PyCharm and VS Code are the most popular. For data science, Spyder is widely used. For beginners, Thonny or IDLE are great choices.

The post Top IDEs for Python developers appeared first on Customer Experience Management.

In this tutorial, we’ll show you exactly how to do that. We’ll build a powerful yet simple Python script that automatically scans a webpage, finds all the PDF links, and searches for specific text inside each one. Using popular libraries like Requests, BeautifulSoup, and PyPDF, you’ll learn a practical skill that can save you hours of manual work. Let’s get started!

Python, Web Scraping, PDF, Automation, BeautifulSoup, PyPDF, requests, Data Extraction, Python Projects, Text Search

Scrape a Website

in the script we Find all links on the initial page. Filter for links that end with .pdf. For each PDF link: Download the PDF file into memory. Extract text from every page of the PDF. Search the extracted text for your search_string. And finally Report which PDF files contain the phrase. Scrape a Website

import requests
from bs4 import BeautifulSoup
from pypdf import PdfReader
import io

Scrape a Website
def find_linked_pdfs(url):
    """
    Scans a webpage for PDF links and searches for a string within each PDF.

    Args:
        url: The URL of the webpage to scan.
        search_string: The string to search for inside the PDFs.
    """
    print(f"Scanning {url} for PDF links...")
    try:
        # 1. Get the main page to find all links
        base_url_parts = requests.utils.urlparse(url)
        base_url = f"{base_url_parts.scheme}://{base_url_parts.netloc}"
        
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')

        pdf_links = [a['href'] \
          for a in soup.find_all('a', href=True) \
          if a['href'].endswith('.pdf')]
        
        if not pdf_links:
            print("No PDF links found on the page.")
            return

        print(f"Found {len(pdf_links)} PDF files. Now searching inside them...")

    return pdf_links

import requests
from bs4 import BeautifulSoup
from pypdf import PdfReader
import io

Scrape a Website
def find_linked_pdfs(url):
    """
    Scans a webpage for PDF links and searches for a string within each PDF.

    Args:
        url: The URL of the webpage to scan.
        search_string: The string to search for inside the PDFs.
    """
    print(f"Scanning {url} for PDF links...")
    try:
        # 1. Get the main page to find all links
        base_url_parts = requests.utils.urlparse(url)
        base_url = f"{base_url_parts.scheme}://{base_url_parts.netloc}"
        
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')

        pdf_links = [a['href'] \
          for a in soup.find_all('a', href=True) \
          if a['href'].endswith('.pdf')]
        
        if not pdf_links:
            print("No PDF links found on the page.")
            return

        print(f"Found {len(pdf_links)} PDF files. Now searching inside them...")

    return pdf_links

Handling URLs: It constructs a full, absolute URL for each PDF, as many links on a page can be relative (e.g., /path/to/file.pdf). Scrape a Website. https://pypi.org/project/beautifulsoup4/

In-Memory Processing: Instead of saving each PDF to your disk, it uses io.BytesIO to treat the downloaded content as a file in your computer’s memory. This is faster and cleaner.

Text Extraction: The pypdf library’s PdfReader opens this in-memory file. The script then loops through each page, calls extract_text(), and combines the text from all pages.

Searching and Reporting: Finally, it performs a case-insensitive search on the extracted text and prints the URL of any PDF that contains your search term.

Search Inside PDF
def find_text_in_pdfs(pdf_links, search_string):

        # 2. Loop through each PDF link

        found_in_files = []
        for pdf_path in pdf_links:
            # Construct absolute URL if the link is relative
            if not pdf_path.startswith(('http://', 'https://')):
                pdf_url = f"{base_url}{pdf_path}"
            else:
                pdf_url = pdf_path

            try:
                # 3. Download the PDF content
                pdf_response = requests.get(pdf_url)
                pdf_response.raise_for_status()

                # Use an in-memory buffer to read the PDF
                pdf_file = io.BytesIO(pdf_response.content)
                reader = PdfReader(pdf_file)
                
                # 4. Extract text and search
                full_text = ""
                for page in reader.pages:
                    full_text += page.extract_text() or ""
                
                if search_string.lower() in full_text.lower():
                    print(f"✔️ Found '{search_string}' in: {pdf_url}")
                    found_in_files.append(pdf_url)

            except Exception as e:
                print(f"⚠️ Could not process {pdf_url}. Reason: {e}")
        
        if not found_in_files:
            print(f"\nSearch complete. The string '{search_string}' was not found in any of the PDFs.")

    except requests.exceptions.RequestException as e:
        print(f"An error occurred fetching the main URL: {e}")

Search Inside PDF
def find_text_in_pdfs(pdf_links, search_string):

        # 2. Loop through each PDF link

        found_in_files = []
        for pdf_path in pdf_links:
            # Construct absolute URL if the link is relative
            if not pdf_path.startswith(('http://', 'https://')):
                pdf_url = f"{base_url}{pdf_path}"
            else:
                pdf_url = pdf_path

            try:
                # 3. Download the PDF content
                pdf_response = requests.get(pdf_url)
                pdf_response.raise_for_status()

                # Use an in-memory buffer to read the PDF
                pdf_file = io.BytesIO(pdf_response.content)
                reader = PdfReader(pdf_file)
                
                # 4. Extract text and search
                full_text = ""
                for page in reader.pages:
                    full_text += page.extract_text() or ""
                
                if search_string.lower() in full_text.lower():
                    print(f"✔️ Found '{search_string}' in: {pdf_url}")
                    found_in_files.append(pdf_url)

            except Exception as e:
                print(f"⚠️ Could not process {pdf_url}. Reason: {e}")
        
        if not found_in_files:
            print(f"\nSearch complete. The string '{search_string}' was not found in any of the PDFs.")

    except requests.exceptions.RequestException as e:
        print(f"An error occurred fetching the main URL: {e}")

x

if __name__ == "__main__":
    target_url = "https://www.umcs.pl/pl/plany-zajec,10795.htm"
    search_term = "programming"
    pdf_links = find_linked_pdfs(target_url)
    find_text_in_pdfs(pdf_links, search_string)
    

if __name__ == "__main__":
    target_url = "https://www.umcs.pl/pl/plany-zajec,10795.htm"
    search_term = "programming"
    pdf_links = find_linked_pdfs(target_url)
    find_text_in_pdfs(pdf_links, search_string)
    

Wrapping Up and Next Steps

Congratulations! You’ve successfully built a powerful automation script that bridges the gap between web scraping and document analysis. By combining the strengths of Requests, BeautifulSoup, and PyPDF, you can now programmatically find information that was previously locked away inside PDF files on any website. This not only saves an incredible amount of time but also opens up new possibilities for data collection and analysis. Feel free to adapt the code for your own projects and take your web scraping skills to the next level. Scrape a Website.

The applications for this technique extend far beyond a single use case. Imagine using this script for academic research, automatically scanning university archives for papers mentioning a specific topic. You could adapt it for financial analysis by pulling keywords from dozens of quarterly earnings reports, or for legal work by searching through court filings for a particular case name. Job seekers could even use it to scan company websites for PDF job descriptions that contain key skills.

To perform a statistical analysis of the overall economy, you can leverage a variety of online resources, including government and intergovernmental data portals, as well as academic publications. These sources often provide data in structured formats like CSVs and APIs, but also in less-structured formats like HTML tables and PDFs, which can be parsed using Python libraries like Beautiful Soup and pypdf.

Government and Intergovernmental Data Sources

For raw, official economic data, these are your most reliable sources. They offer a wealth of information on everything from GDP and inflation to employment rates and international trade. Scrape a Website. Search Inside PDF

• Federal Reserve Economic Data (FRED): A fantastic resource from the St. Louis Fed, FRED offers over 800,000 economic time series from more than 100 sources. It’s a goldmine for anyone doing macroeconomic analysis.
• The World Bank Open Data: This portal provides comprehensive global development data, including indicators on economic policy, poverty, gender, and more, making it perfect for cross-country comparisons.
• Data.gov: The home of U.S. government open data, this site aggregates datasets from various federal agencies, including the Bureau of Economic Analysis (BEA) and the Bureau of Labor Statistics (BLS).
• United Nations Statistics Division (UNSD): The UNSD offers a wide array of international statistics, including the UNdata portal which provides free access to over 60 million statistical records from various UN agencies.
• The Bureau of Economic Analysis (BEA): The BEA produces some of the most critical U.S. economic statistics, such as GDP, personal income, and corporate profits.

You can read about Business analyst carrier path here

The core principle remains the same: automate the discovery of information, no matter the format. Search Inside PDF. Happy coding! 🚀

The post How to Scrape a Website and Search Inside PDFs with Python appeared first on Customer Experience Management.

Attribution Models and Credit

Google Ads Attribution: By default, Google Ads uses a data-driven attribution model, which gives credit to various touchpoints along the conversion path. By default ist is 30 days. It will take credit for a conversion as long as a user interacted with one of your ads within 30 days. You can defined shorter lookback window.

GA4 Attribution: GA4’s default is also data-driven, but it’s “cross-channel.” This means it considers all marketing channels (organic search, direct, social, email, etc.) in the customer journey, not just Google Ads. If a user clicks a Google ad, then later comes back to your site via organic search and converts, GA4’s data-driven model will distribute credit to both channels, while Google Ads will likely claim all or most of the conversion value.

Time-Based Reporting Google Ads Conversion and GA4 Revenue Difference

The way each platform logs a conversion can create a significant reporting gap, especially when analyzing recent data.

• Google Ads: Attributes a conversion to the date of the ad click or impression.
• GA4: Attributes a conversion to the date of the actual transaction or conversion event.

For example, if a user clicks a Google ad on September 20th but makes a purchase on October 5th, Google Ads will report the conversion value in September, while GA4 will report it in October. This “conversion lag” can cause large differences when comparing monthly or weekly reports. Google Ads Conversion and GA4 Revenue Difference.

Conversion Counting and Definitions

How you set up your conversion counting and conversion values can lead to different numbers. Especially important is conversion values for e-commerce purchase.

Conversion Counting: In Google Ads, you can choose to count a conversion “once” or “every” time it happens. For example, if a user submits a form multiple times, you might want to count it only once. However, for a purchase, you’d want to count every transaction. If these settings are not aligned between Google Ads and GA4, your numbers will not match.

Conversion Actions: If you have different conversion actions set up in Google Ads and GA4, or if they are not correctly linked, you will see a discrepancy. For example, if you track a “purchase” in GA4 but have a different conversion action for “leads” in Google Ads, the numbers will naturally be different.

Technical and User-Based Factors

These are often smaller but can add up to a substantial difference.

• Ad Blockers and User Consent: Some ad blockers and privacy settings can prevent GA4’s tracking code from firing, meaning a session and conversion might not be recorded in GA4. However, the Google Ads conversion tag is often less affected, so Google Ads may still report the conversion. Similarly, if a user opts out of tracking via a consent banner, GA4 may not receive the data.
• Quick Exits: A user may click a Google ad, be charged for the click, and then hit the back button before the GA4 tracking tag has a chance to load. Google Ads will count the click, but GA4 won’t record a session, leading to a discrepancy between clicks and sessions, and ultimately, conversions.
• Cross-Device Conversions: Google Ads uses modeled conversions to account for users who start their journey on one device and finish it on another. This can lead to a higher conversion count in Google Ads than in GA4, especially if Google Signals is not enabled in GA4.
• View-Through Conversions: Google Ads counts “view-through conversions,” which are conversions that happen after a user sees a display ad but doesn’t click on it. GA4 does not track these by default, which can cause Google Ads to report more conversions and higher conversion value.

What to Do About It

Align Your Attribution Models: Consider using the same attribution model in both platforms for a more direct comparison. While Google Ads defaults to data-driven, you can use the “Model Comparison” tool in GA4 to see how your data would look under a different model, like “Last Click.”

The Model Comparison report, also referred to as the Attribution models report, in Google Analytics 4 (GA4) is a tool that allows you to compare how different attribution models distribute credit for conversions. It helps you understand how various marketing channels, like paid search, social media, and organic search, contribute to a user’s conversion path. This is a crucial report for understanding the true value of your marketing efforts beyond just the last touchpoint.

The report lets you select a conversion event and then view how its value would be allocated to different channels under two different attribution models. You can then see a percentage change, which highlights which channels are being over or undervalued depending on the model you use.

Key Attribution Models to Compare

GA4 offers a few attribution models to choose from, with Data-driven being the default. Comparing this with other models is where the tool’s value really shines.

• Data-driven: This model uses machine learning to analyze all the touchpoints in a user’s journey, including both converting and non-converting paths. It then assigns credit based on the actual impact of each touchpoint. It’s considered the most accurate and sophisticated model.
• Paid and organic last click: This is a rule-based model that gives 100% of the conversion credit to the last channel a user clicked on before converting, ignoring any direct traffic.
• Google paid channels last click: This model gives 100% of the credit to the last Google Ads click before a conversion. If there was no Google Ads click, it defaults to the “Paid and organic last click” model.

Why is this important?

Understanding how different attribution models impact your data is critical for making smart business decisions. For instance, a “last click” model might make your paid search campaigns look like they’re driving all your conversions, leading you to undervalue channels like social media or display ads that introduce users to your brand much earlier in their journey. By using the Model Comparison report, you can see how a channel’s value changes when you shift from a last-click to a data-driven model, which can lead to better budgeting and optimization.

The video below offers a tutorial on how to use the Model Comparison Tool to analyze your traffic channels and conversion data in Google Analytics. How to Use the Model Comparison Tool in Google Analytics to Compare Your Traffic Channels

Check Your Conversion Settings: Ensure that your conversion actions are correctly set up and linked, and that the counting method is consistent across both platforms.

Check Your Date Ranges: When comparing, make sure you’re using a long enough date range (e.g., a full month or longer) to account for any conversion lag.

Enable Auto-Tagging: Make sure auto-tagging is enabled in your Google Ads account so that GA4 can properly attribute traffic back to your campaigns.

Don’t Panic: It’s normal to have some level of discrepancy. The key is to understand why the differences exist and use both platforms for what they’re best at: Google Ads for optimizing your paid campaigns and GA4 for understanding the full, multi-channel customer journey.

Read more here

The post Google Ads Conversion and GA4 Revenue Difference appeared first on Customer Experience Management.

Python’s powerful date and time functions using the datetime and pandas libraries gives you a robust date table ready for Power BI and other business intelligence and analytical tools. Python for analysts most important datetime functions.

Mastering Date and Time Functions in Python for Power BI Date Tables

When working with Power BI, a well-structured Date Table is essential for time intelligence calculations like YTD, QTD, MTD, and custom period comparisons. While Power BI has built-in date table features, using Python to generate a custom date table gives you full control over the structure, granularity, and logic.

In this post, we’ll explore Python’s powerful date and time functions using the datetime and pandas libraries, and show how to create a robust date table ready for Power BI.

Python for analyst – date and time functions – basics

Python provides the datetime module to work with dates and times. Here’s a quick overview:

from datetime import datetime, timedelta, date

# Current date and time
now = datetime.now()
print("Now:", now)

# Just the date
today = date.today()
print("Today:", today)

# Add 7 days
next_week = today + timedelta(days=7)
print("Next week:", next_week)

# Subtract 30 days
last_month = today - timedelta(days=30)
print("30 days ago:", last_month)

These functions are the foundation for generating date ranges and calculating custom columns like fiscal periods or holidays. Python for analysts most important datetime functions.

Creating a Date Range with Pandas

To build a date table, we need a continuous range of dates. pandas.date_range() is perfect for this:

import pandas as pd

# Generate a date range from 2020 to 2030
date_range = pd.date_range(start='2020-01-01', end='2030-12-31', freq='D')
df = pd.DataFrame({'Date': date_range})

This gives us a DataFrame with one row per day — the backbone of our date table.

Enriching the Date Table

Now let’s add useful columns for Power BI:

df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['MonthName'] = df['Date'].dt.strftime('%B')
df['Quarter'] = df['Date'].dt.quarter
df['Day'] = df['Date'].dt.day
df['Weekday'] = df['Date'].dt.weekday + 1  # Monday = 1
df['WeekdayName'] = df['Date'].dt.strftime('%A')
df['IsWeekend'] = df['Weekday'].isin([6, 7])
df['Week'] = df['Date'].dt.isocalendar().week
df['DayOfYear'] = df['Date'].dt.dayofyear

These columns allow for slicing and dicing your data in Power BI by year, month, weekday, and more.

Fiscal Calendar Support

Many businesses use fiscal calendars that don’t align with the calendar year. Here’s how to add a fiscal year starting in July:

df['FiscalYear'] = df['Date'].apply(lambda x: x.year if x.month < 7 else x.year + 1)
df['FiscalMonth'] = df['Date'].apply(lambda x: x.month - 6 if x.month >= 7 else x.month + 6)
df['FiscalQuarter'] = ((df['FiscalMonth'] - 1) // 3) + 1

This logic adjusts the fiscal year, month, and quarter based on a July start. Python for analysts most important datetime functions

5. Flags for Time Intelligence

Power BI benefits from flags that simplify DAX calculations:

today = pd.to_datetime('today').normalize()

df['IsToday'] = df['Date'] == today
df['IsCurrentMonth'] = (df['Date'].dt.month == today.month) & (df['Date'].dt.year == today.year)
df['IsCurrentYear'] = df['Date'].dt.year == today.year

You can also add flags for holidays, fiscal periods, or custom business logic. Python for analysts most important datetime functions

Exporting to CSV for Power BI

Once your date table is ready, export it:

df.to_csv('DateTable.csv', index=False)

You can now import this CSV into Power BI as a static date table.

7. Sample Output

Here’s a preview of what your date table might look like:

Pandas to_datetime() function

Python for analysts most important datetime functions – pandas

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  object        
 3   Dt_last_purchase        13483 non-null  object        

df_cust['Dt_first_purchase'] = pd.to_datetime(df_cust['Dt_first_purchase'],format="yyyy-mm-dd")
df_cust['Dt_last_purchase'] = pd.to_datetime(df_cust['Dt_last_purchase'],format="yyyy-mm-dd")

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  datetime64[ns]
 3   Dt_last_purchase        13483 non-null  datetime64[ns]

Advanced Tips

• Holidays: Use external APIs or CSVs to mark public holidays.
• Week Start: Adjust Weekday to match your locale (e.g., Monday vs. Sunday).
• Time Zones: Use pytz or zoneinfo for timezone-aware datetime handling.
• Dynamic Updates: Automate the script to regenerate the table monthly or yearly.

Certainly! Here’s a concise 400-word post on SQL Date and Time Functions, with examples, tailored for building a Date Table in Power BI:

SQL Date and Time Functions for Power BI Date Tables

When building reports in Power BI, a comprehensive Date Table is essential for enabling time-based calculations like YTD, MTD, and custom period comparisons. While Power BI can auto-generate a date table, using SQL to create one gives you full control over its structure and logic.

Let’s explore key SQL Server date and time functions and how to use them to build a robust date table.

1. Generating a Date Range

To create a date table, you need a continuous range of dates. In SQL Server, you can use a loop or a recursive CTE:

DECLARE @StartDate DATE = '2020-01-01';
DECLARE @EndDate DATE = '2030-12-31';

WITH DateCTE AS (
    SELECT @StartDate AS DateValue
    UNION ALL
    SELECT DATEADD(DAY, 1, DateValue)
    FROM DateCTE
    WHERE DateValue < @EndDate
)
SELECT * INTO DateTable FROM DateCTE
OPTION (MAXRECURSION 32767);

select * from DateTable

Here the example

This creates a table with one row per day between 2020 and 2030.

2. Adding Date Attributes

Once you have the base dates, enrich them with useful columns:

ALTER TABLE DateTable ADD 
    Year INT,
    Month INT,
    MonthName VARCHAR(20),
    Quarter INT,
    Weekday INT,
    WeekdayName VARCHAR(20);

UPDATE DateTable
SET 
    Year = YEAR(DateValue),
    Month = MONTH(DateValue),
    MonthName = DATENAME(MONTH, DateValue),
    Quarter = DATEPART(QUARTER, DateValue),
    Weekday = DATEPART(WEEKDAY, DateValue),
    WeekdayName = DATENAME(WEEKDAY, DateValue);

These columns allow for flexible filtering and grouping in Power BI.

3. Fiscal Calendar and Flags

You can also add fiscal logic and flags:

ALTER TABLE DateTable ADD FiscalYear INT;

UPDATE DateTable
SET FiscalYear = CASE 
    WHEN MONTH(DateValue) >= 7 THEN YEAR(DateValue) + 1
    ELSE YEAR(DateValue)
END;

Add flags like IsWeekend, IsToday, or IsCurrentMonth to simplify DAX expressions.

Conclusion

Python offers a flexible and powerful way to create a custom date table for Power BI. With just a few lines of code, you can generate a rich dataset that supports advanced time intelligence and reporting needs.

Whether you’re working with fiscal calendars, custom flags, or multilingual support, Python gives you the tools to tailor your date table exactly to your business requirements.

SQL’s date and time functions like DATEADD, DATEPART, DATENAME, and YEAR are powerful tools for building a custom date table. Once created, export it to Power BI or use it as a view for dynamic reporting.

Would you like a ready-to-run SQL script for a complete date table?

Find more resources in our course of Advanced programming for business analysts

The post Python for analysts most important datetime functions appeared first on Customer Experience Management.

Cel tego wydarzenia

Ośrodek Sportu i Rekreacji w Biłgoraju “Na Fali” ma zaszczyt zaprosić na wyjątkowe wydarzenie, które po raz drugi zjednoczy nas wokół szczytnego celu! W dniach 25-27 lipca 2025 roku Zalew Bojary w Biłgoraju stanie się areną drugiej edycji akcji “Na Fali Nadziei – Przekraczając Granice”! – czytaj tutaj

W tym roku wspieramy Adasia Iwanejko, 12-latka z Woli Małej, który dzielnie walczy ze śmiertelną dystrofią mięśniową Duchenne’a. Jego jedyną szansą na powrót do zdrowia jest kosztowna terapia genowa w USA, której szacunkowy koszt przekracza 16 milionów złotych. Wierzymy, że dzięki wspólnej mobilizacji możemy zdziałać cuda i pomóc Adasiowi spełnić marzenie o normalnym życiu!

Zadanie 1. Biłgoraj – Zalew Bojary – pływanie 100 km

Pływak Damian Błaszczyk ma pokonać dystans 100 km pływając po okręgu przywiązany na linie o długości 100m do słupa o nieistotnej grubości. Lina obraca się wokół słupa zachowująć cały czas długość 100m. Zadanie to jest do wykonania na zalewie Bojary ponieważ zalew ten ma średnicę około 230m. Tak więc zakładamy, że pływak będzie okrążał zalew aż pokona dystans D=100km. Pytanie jest ile okrążeń musi wykonać pływak?

Obliczenia przedstawiono w tabeli poniżej. Pływak wykona 8 rund po 10 okrążeń w prawo, co daje 80 okrążeń i 80 okrążeń w lewo, co daje razem 160 okrążeń. Każde okrążenie daje dystans d=2Pir = 628 m. Tak więc jedna runda 10 okrążeń daje 6,280 m, 8 rund to 50,240 m. Przepłynięcie 100 km wymaga więc 16 rund czyli 160 okrążeń zalewu. Płynąc w tempie 4 km/h zajmuje czas około 25 godzin. W informacji prasowej jest wzmianka, że pływak zamierza pływać 48 godzin, więc jest to realny czas na pokonanie dystansu 100 km. Biłgoraj – Zalew Bojary – pływanie 100 km

Zadanie 2

Teraz zakładamy, że średnicy słupa wynosi 20 cm, co jest bardziej realne. Lina nawija się na słup w ten sposób, że po każdych 10 ciu okrążeniach przeskakuje na kolejną warstwę zwiększając tym samym średnicę słupa. Pytanie: ile okrążeń musi wykonać pływak aby pokonać dystans 100 km? Biłgoraj – Zalew Bojary – pływanie 100 km.

Rozwiązanie zadania:

• Długość początkowa liny = promień początkowy okręgu = 100 m.
• Średnica liny = 1 cm ( rl = 0,01 m).
• Średnica słupa = 20 cm (promień słupa = 10cm = r0 = 0,1m).

Etap 1, Pływak w pierwszym okrążeniu pokona dystans d = 2Pir = 628 m. W czasie tego okrążenia lina nawinie się na słup, co spowoduje, że jej długość zmniejszy się o d0 = 2Pi * r0 = 0,628 m. Tak więc drugie okrążenie będzie miało długość 628 m – 0,628 m = 627,2 m. Itd aż do 10-tego okrążenia. Czyli w okrążeniach 1-10, lina skraca się o d0 = 2*Pi*r0 = 6.28 * 0,1m = 0,628 m per okrążenie, czyli razem skróci się o 6,28m. Po 10 okrążeniach lina przeskakuje na wierzch poprzednio nawiniętej warstwy, co zwiększa promień słupa o grubość liny, czyli o 1 cm (0,01 m). Tak więc w okrążeniach 11-20, lina skraca się o d2 = 2 Pi * (r0+rl) = 6.28 * 0,11 m = 0,69 m.

Całość obliczeń przedstawia tabela poniżej. Pływak bedzie płynął na coraz to krótszej linie, co spowoduje, że ostatecznie cała lina zawinie się na słup. Po ilu okrążeniach to nastąpi i jaki dystans pływak pokona w tym czasie?

Jeśli pierwsze 10 okrążeń skraca line o 6,28, drugie 10 okrążeń skraca linę o 6,91 m itd. – patrz kolumnę czwartą Ubytek*10, to 100 m liny wystarczy na 10 rund (etapów) po 10 okrążeń i 11 rundę 6 okrążeń. Razem będzie to 106 okrążeń. Da to dystans 37,655 m. Jeśli pływak płynął w prawo to teraz powinien płynąć w lewo, co da kolejne 37,655 m. Ciągle nie będzie to pełne 100 km. Potrzebne są jeszcze 4 rundy o 8 okrążeń. Czyli podsumowując: pływak przepłynie łącznie 106 okrążeń w prawo i 106 w lewo a potem jeszcze 48 okrążeń w prawo. Razem da to 260 okrążeń. Z racji tego, że promień okręgu jest zmienny, co powoduje, że pokonywany dystans jest zmienny z każdym okrążeniem, to do pokonania 100 km potrzebne jest 260 okrążeń czyli o 100 wiecej niż w poprzedniej metodzie. Biłgoraj – Zalew Bojary – pływanie 100 km.

Zadanie 3

Ciekawe zadanie powstaje, jeśli lina będzie się skracać za każdym razem nawijając się na poprzednie warstwy tworząc spiralę. W takiej spirali każde okrążenie jest krótsze o 0,628 m tworząc ciąg liczbowy. Do rozwiązanie tego zadania wykorzystaliśmy Google Gemini https://gemini.google.com , co przedstawia poniższy obrazek:

Równanie to można uprościć do następującej formy ( Biłgoraj – Zalew Bojary – pływanie 100 km )

i następnie

co po wymnożeniu daje równanie kwadratowe

i następnie

W tej sytuacji mamy równanie kwadratowe typu ax^2 + bx + c =0. Używając formuły kwadratowej x = (−b ± pierwiastek kwadratowy z ( b2 − 4ac) ) / 2a​​, dostajemy: n = 160,43. Tak więc w takiej sytuacji pływak wykona 160 okrążenia na ciągle skracającej się linie. Biłgoraj – Zalew Bojary – pływanie 100 km

Dla miłośników Pythona, zadanie może być rozwiązane przy pomocy poniższej procedury. Biłgoraj – Zalew Bojary – pływanie 100 km.

def calculate_turns(rope_length, rope_thickness, bar_radius, shorten_per_turn, turns_per_layer, layer_increase):
    turns = 0
    while rope_length > 0:
        turns += 1
        rope_length -= shorten_per_turn
        if turns % turns_per_layer == 0:
            bar_radius += layer_increase / 2
            shorten_per_turn = 2 * 3.14 * (bar_radius + rope_thickness)
    return turns
# Given values
rope_length = 100  # meters
rope_thickness = 0.01  # meters (1 cm)
bar_radius = 0.1  # meters
shorten_per_turn = 0.628  # meters
turns_per_layer = 10
layer_increase = 0.02  # meters (2 cm)

# Calculate the number of turns
turns = calculate_turns(rope_length, rope_thickness, bar_radius, shorten_per_turn, turns_per_layer, layer_increase)

print(f"The swimmer can make {turns} turns before the rope ends.")

More python here

Biłgoraj – Zalew Bojary – pływanie 100 km

Pływania w dobrej sprawie – Biłgoraj – Zalew Bojary – pływanie 100 km

Artykuł opisuje niezwykłe wyzwanie wytrzymałościowe podjęte przez pływaka Damiana Błaszczyka, który postanowił przepłynąć 100 kilometrów w zbiorniku wodnym Zalew Bojary w Biłgoraju. Wydarzenie to, będące częścią inicjatywy „Na Fali Nadziei – Przekraczając Granice”, ma nie tylko sportowy, ale przede wszystkim charytatywny charakter. Celem tegorocznej edycji jest wsparcie Adasia Iwanejko, 12-letniego chłopca chorego na dystrofię mięśniową Duchenne’a. Zebrane środki mają pomóc w sfinansowaniu kosztownej terapii genowej w USA, której koszt przekracza 16 milionów złotych.

Matematyka pływania – modelowanie dystansu

Autor bloga przedstawia matematyczne modele opisujące, jak można przepłynąć 100 km w jednym zbiorniku. W pierwszym scenariuszu Damian pływa po idealnym okręgu, przymocowany do słupa liną o długości 100 metrów. Każde okrążenie ma wtedy około 628 metrów (obliczone ze wzoru na obwód koła: (2\pi r)). Aby osiągnąć 100 km, potrzeba 160 okrążeń, zmieniając kierunek co 10, by równomiernie obciążać ciało. Przy stałym tempie 4 km/h, całość zajęłaby około 25 godzin, choć wydarzenie przewiduje 48 godzin, co czyni cel realnym.

Realizm: lina nawijająca się na słup

Drugi scenariusz uwzględnia bardziej realistyczny aspekt – lina nawija się na słup, skracając z każdym okrążeniem promień pływania. Słup ma 20 cm średnicy, a lina 1 cm grubości. Po każdych 10 okrążeniach lina tworzy nową warstwę, zwiększając promień słupa. W efekcie każde kolejne okrążenie jest krótsze. Obliczenia pokazują, że Damian może wykonać 106 okrążeń w jednym kierunku, pokonując 37,7 km. Powtórzenie tego w przeciwnym kierunku daje kolejne 37,7 km. Aby osiągnąć 100 km, potrzeba jeszcze 48 dodatkowych okrążeń, co daje łącznie 260 okrążeń – o 100 więcej niż w pierwszym modelu.

Model spiralny – matematyczna elegancja

Trzeci model zakłada spiralne nawijanie liny, gdzie każde okrążenie jest krótsze o stałą wartość. Tworzy to ciąg arytmetyczny długości okrążeń. Korzystając z wzoru na sumę ciągu, autor oblicza, że potrzeba około 160,43 okrążeń, by osiągnąć 100 km. Model ten jest zbliżony do pierwszego, ale bardziej realistyczny i matematycznie elegancki.

Podsumowanie

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. Biłgoraj – Zalew Bojary – pływanie 100 km.

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. To także przykład, jak pasja i nauka mogą wspólnie służyć wyższemu celowi.

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. To także przykład, jak pasja i nauka mogą wspólnie służyć wyższemu celowi.

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. To także przykład, jak pasja i nauka mogą wspólnie służyć wyższemu celowi.

The post Biłgoraj – Zalew Bojary – pływanie 100 km appeared first on Customer Experience Management.

How AI is changing search behaviors

https://www.nngroup.com/articles/ai-changing-search-behaviors

Semantic search with Elasticsearch – intro

https://www.elastic.co/search-labs/blog/introduction-to-vector-search

Language model implementation

As the first we should initiate a language model and prepare product data for input. Semantic search with Elasticsearch require data indexing via vector transformer.

import pandas as pd
import numpy as np

# function to normalize vectors
def normalize_embedding(embedding):
    norm = np.linalg.norm(embedding)
    return embedding / norm if norm != 0 else embedding

# select products from sql database
def select_products():
    q = """
        SELECT 
         [ProductId]
        ,[ProdIdx]
        ,[ProductName]
        FROM [DB_Products] 
    """
    dfp = read_from_sql_server(q);
    return dfp

# import sentence transformers and initiate model
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
print(model)

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

model = model.to(device)
print(model)

import pandas as pd
import numpy as np

# function to normalize vectors
def normalize_embedding(embedding):
    norm = np.linalg.norm(embedding)
    return embedding / norm if norm != 0 else embedding

# select products from sql database
def select_products():
    q = """
        SELECT 
         [ProductId]
        ,[ProdIdx]
        ,[ProductName]
        FROM [DB_Products] 
    """
    dfp = read_from_sql_server(q);
    return dfp

# import sentence transformers and initiate model
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
print(model)

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

model = model.to(device)
print(model)

Data indexing for semantic search

Now we can index product data into Elasticsearch database (index). We will index product names as vectors and lexically. This allows hybrid search. Semantic search with Elasticsearch.

# initiate Elasticsearch client
from elasticsearch import Elasticsearch, helpers
es = Elasticsearch('http://localhost:9100')

# check es client
from pprint import pprint
pprint(es.info().body)

# ReCreate the index with dense_vector and text mappings
# step 1
es.indices.delete(index="prod_search_hybrid", ignore_unavailable=True)

# step 2 - mappings
es.indices.create(
    index="prod_search_hybrid",
    mappings={
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims": 384,
                "index": True,
                "similarity": "cosine"
            },
            "prodct_name": {
                "type": "text"
            }
        }
    },
)

# list indices and number of documents indexed 
def indices_list():
    indices = es.cat.indices(format='json')
    return [x['index'] for x in indices]
# -------------------
print(indices_list())
# ------------------------------

# Create documents for embedding
documents = []
for i, r in df_docs.iterrows():
    documents.append({        
        'product_name': r['ProductName'][:256].lower(),        
    })

print(f' Created table of {len(documents)} docs')

# Prepare bulk operations
from tqdm import tqdm                                         # for a prograss bar
operations = []
for document in tqdm(documents, total=len(documents)):
    operations.append({'index': {'_index': 'prod_search_hybrid'}})
    operations.append({
        **document,
        'embedding': get_embedding(document['product_name']), # vectors for semantic search
        'product_name': document['product_name']              # the text field for hybrid search
    })

# Bulk insert the data into Elasticsearch
response = es.bulk(operations=operations)
print(f' Records indexed: {len(response["items"])}')

# initiate Elasticsearch client
from elasticsearch import Elasticsearch, helpers
es = Elasticsearch('http://localhost:9100')

# check es client
from pprint import pprint
pprint(es.info().body)

# ReCreate the index with dense_vector and text mappings
# step 1
es.indices.delete(index="prod_search_hybrid", ignore_unavailable=True)

# step 2 - mappings
es.indices.create(
    index="prod_search_hybrid",
    mappings={
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims": 384,
                "index": True,
                "similarity": "cosine"
            },
            "prodct_name": {
                "type": "text"
            }
        }
    },
)

# list indices and number of documents indexed 
def indices_list():
    indices = es.cat.indices(format='json')
    return [x['index'] for x in indices]
# -------------------
print(indices_list())
# ------------------------------

# Create documents for embedding
documents = []
for i, r in df_docs.iterrows():
    documents.append({        
        'product_name': r['ProductName'][:256].lower(),        
    })

print(f' Created table of {len(documents)} docs')

# Prepare bulk operations
from tqdm import tqdm                                         # for a prograss bar
operations = []
for document in tqdm(documents, total=len(documents)):
    operations.append({'index': {'_index': 'prod_search_hybrid'}})
    operations.append({
        **document,
        'embedding': get_embedding(document['product_name']), # vectors for semantic search
        'product_name': document['product_name']              # the text field for hybrid search
    })

# Bulk insert the data into Elasticsearch
response = es.bulk(operations=operations)
print(f' Records indexed: {len(response["items"])}')

Hybrid search

For hybrid search we combine match and knn search inside a bool query. The _name field return what what part of the query has returned results. This allow to build hybride scoring. As you can see we vectorize query_h to query_vector using the same procedure get_embeding as we were using during indexing. Semantic search with Elasticsearch.

query_h = "anti-explosion device"

# Print the query vector for debugging
# query_vector = get_embedding(query_h)
# print("Query Vector:", query_vector)

response = es.search(
    index='prod_search_hybrid',
    body={
        "query": {
            "bool": {
                "should": [
                    {
                        "match": {
                            "product_name": {
                                "query": query_h,
                                "_name": "text_match"
                            }
                        }
                    },
                    {
                        "knn": {
                            "field": "embedding",
                            "query_vector":  query_vector,
                            "k": 10,
                            "num_candidates": 100,
                            "_name": "semantic_search"
                        }
                    }
                ]
            }
        }
        ,
        'size': 30
    }
)

query_h = "anti-explosion device"

# Print the query vector for debugging
# query_vector = get_embedding(query_h)
# print("Query Vector:", query_vector)

response = es.search(
    index='prod_search_hybrid',
    body={
        "query": {
            "bool": {
                "should": [
                    {
                        "match": {
                            "product_name": {
                                "query": query_h,
                                "_name": "text_match"
                            }
                        }
                    },
                    {
                        "knn": {
                            "field": "embedding",
                            "query_vector":  query_vector,
                            "k": 10,
                            "num_candidates": 100,
                            "_name": "semantic_search"
                        }
                    }
                ]
            }
        }
        ,
        'size': 30
    }
)

After that we can extract product names and scoring and build a list according to our intention. In here we separate products according to field _name and then build list of top 10 lexical match and semantic similarity. More about similarity measures you can read in post Measuring product similarity

# Extract product names, scores, and sources
products = [
    (hit["_source"]["product_name"], hit["_score"], hit.get("matched_queries", []))
    for hit in response["hits"]["hits"]
]

# Separate products into text_match and semantic_search groups
text_match_products = [product for product in products if "text_match" in product[2]]
semantic_search_products = [product for product in products if "semantic_search" in product[2]]

# Sort each group by score in descending order
sorted_text_match_products = sorted(text_match_products, key=lambda x: x[1], reverse=True)[:10]
sorted_semantic_search_products = sorted(semantic_search_products, key=lambda x: x[1], reverse=True)[:10]

# Print top 10 text_match products
print("\nTop 10 Text Match Products:")
for product in sorted_text_match_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

# Print top 10 semantic_search products
print("\nTop 10 Semantic Search Products:")
for product in sorted_semantic_search_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

# Extract product names, scores, and sources
products = [
    (hit["_source"]["product_name"], hit["_score"], hit.get("matched_queries", []))
    for hit in response["hits"]["hits"]
]

# Separate products into text_match and semantic_search groups
text_match_products = [product for product in products if "text_match" in product[2]]
semantic_search_products = [product for product in products if "semantic_search" in product[2]]

# Sort each group by score in descending order
sorted_text_match_products = sorted(text_match_products, key=lambda x: x[1], reverse=True)[:10]
sorted_semantic_search_products = sorted(semantic_search_products, key=lambda x: x[1], reverse=True)[:10]

# Print top 10 text_match products
print("\nTop 10 Text Match Products:")
for product in sorted_text_match_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

# Print top 10 semantic_search products
print("\nTop 10 Semantic Search Products:")
for product in sorted_semantic_search_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

Full-text search, also known as lexical search, is a technique for fast, efficient searching through text fields in documents. Documents and search queries are transformed to enable returning relevant results instead of simply exact term matches. Fields of type text are analyzed and indexed for full-text search.

You can combine full-text search with semantic search using vectors to build modern hybrid search applications. While vector search may require additional GPU resources, the full-text component remains cost-effective by leveraging existing CPU infrastructure.

Another example of vector indexing and sementic search you can find here: https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb

Vector search setup and performing hybrid search

https://www.elastic.co/search-labs/blog/vector-search-set-up-elasticsearch

https://www.elastic.co/search-labs/blog/hybrid-search-elasticsearch

Filtering. Semantic search with Elasticsearch

Filter context is mostly used for filtering structured data. For example, use filter context to answer questions like:

• Does this timestamp fall into the range 2015 to 2016?
• Is the status field set to “published”?

Filter context is in effect whenever a query clause is passed to a filter parameter, such as the filter or must_not parameters in a bool query. Learn more about filter context in the Elasticsearch docs.

Keyword Filtering

This is an example of adding a keyword filter to the query. The example retrieves the top books that are similar to “javascript books” based on their title vectors, and also Addison-Wesley as publisher. Semantic search with Elasticsearch.

response = client.search(
    index="book_index",
    knn={
        "field": "title_vector",
        "query_vector": model.encode("javascript books"),
        "k": 10,
        "num_candidates": 100,
        "filter": {"term": {"publisher.keyword": "addison-wesley"}},
    },
)

pprint(response)

Query and filter context

Relevance scores

By default, Elasticsearch sorts matching search results by relevance score, which measures how well each document matches a query. The relevance score is a positive floating point number, returned in the _score metadata field of the search API. The higher the _score, the more relevant the document. While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a query or filter context.

Query context

In the query context, a query clause answers the question How well does this document match this query clause? Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the _score metadata field. Query context is in effect whenever a query clause is passed to a query parameter, such as the query parameter in the search API. Semantic search with Elasticsearch

Filter context

A filter answers the binary question “Does this document match this query clause?”. The answer is simply “yes” or “no”. Filtering has several benefits:

1. Simple binary logic: In a filter context, a query clause determines document matches based on a yes/no criterion, without score calculation.
2. Performance: Because they don’t compute relevance scores, filters execute faster than queries.
3. Caching: Elasticsearch automatically caches frequently used filters, speeding up subsequent search performance.
4. Resource efficiency: Filters consume less CPU resources compared to full-text queries.
5. Query combination: Filters can be combined with scored queries to refine result sets efficiently.

Filters are particularly effective for querying structured data and implementing “must have” criteria in complex searches.

Structured data refers to information that is highly organized and formatted in a predefined manner. In the context of Elasticsearch, this typically includes:

• Numeric fields (integers, floating-point numbers)
• Dates and timestamps
• Boolean values
• Keyword fields (exact match strings)
• Geo-points and geo-shapes

Unlike full-text fields, structured data has a consistent, predictable format, making it ideal for precise filtering operations. Semantic search with Elasticsearch.

Common filter applications include:

• Date range checks: for example is the timestamp field between 2015 and 2016
• Specific field value checks: for example is the status field equal to “published” or is the author field equal to “John Doe”

Filter context applies when a query clause is passed to a filter parameter, such as:

• filter or must_not parameters in bool queries
• filter parameter in constant_score queries
• filter aggregations

Filters optimize query performance and efficiency, especially for structured data queries and when combined with full-text searches.

GET /_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title":   "Search"        }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [
        { "term":  { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}

Read more: Product Search and Product classification for E-commerce

Reciprocal rank fusion

Reciprocal rank fusion (RRF) is a method for combining multiple result sets with different relevance indicators into a single result set. RRF requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results. Semantic search with Elasticsearch.

RRF uses the following formula to determine the score for ranking each document:

score = 0.0
for q in queries:
    if d in result(q):
        score += 1.0 / ( k + rank( result(q), d ) )
return score

# where
# k is a ranking constant
# q is a query in the set of queries
# d is a document in the result set of q
# result(q) is the result set of q
# rank( result(q), d ) is d's rank within the result(q) starting from 1

You can use RRF as part of a search to combine and rank documents using separate sets of top documents (result sets) from a combination of child retrievers using an RRF retriever. A minimum of two child retrievers is required for ranking.

We rank the documents based on the RRF formula with a rank_window_size of 5 truncating the bottom 2 docs in our RRF result set with a size of 3. We end with _id: 3 as _rank: 1, _id: 2 as _rank: 2, and _id: 4 as _rank: 3. This ranking matches the result set from the original RRF search as expected.

In this example, we execute the knn and standard retrievers independently of each other. Then we use the rrf retriever to combine the results.

1. First, we execute the kNN search specified by the knn retriever to get its global top 50 results.
2. Second, we execute the query specified by the standard retriever to get its global top 50 results.
3. Then, on a coordinating node, we combine the kNN search top documents with the query top documents and rank them based on the RRF formula using parameters from the rrf retriever to get the combined top documents using the default size of 10.

Note that if k from a knn search is larger than rank_window_size, the results are truncated to rank_window_size. If k is smaller than rank_window_size, the results are k size.

GET example-index/_search
{
    "retriever": {
        "rrf": {
            "retrievers": [
                {
                    "standard": {
                        "query": {
                            "term": {
                                "text": "shoes"
                            }
                        }
                    }
                },
                {
                    "knn": {
                        "field": "vector",
                        "query_vector": [1.25, 2, 3.5],
                        "k": 50,
                        "num_candidates": 100
                    }
                }
            ],
            "rank_window_size": 50,
            "rank_constant": 20
        }
    }
}

The post Semantic Search with Elasticsearch appeared first on Customer Experience Management.