1. Navigate to the folder that contains all your virtual environments.
2. Run the following command: Bash for venv in */ ; do if [ -f "${venv}bin/pip" ]; then echo "--- Checking in '${venv%?}' ---" ${venv}bin/pip list | grep 'lifelines' fi done

How it works:

• It loops through each subdirectory (e.g., my_project_venv/).
• It checks if a pip executable exists inside the bin folder to confirm it’s likely a venv.
• It then runs the pip list command from within that specific environment and uses grep to filter for the line containing “lifelines”.
• If lifelines is installed, it will print the package name and its version. If not, it will print nothing for that environment.

## For Windows 🪟

You can use a similar loop in either Command Prompt (CMD) or PowerShell.

In PowerShell:

1. Open PowerShell and navigate to the folder containing your virtual environments.
2. Run the following command: PowerShe ll Get-ChildItem -Directory | ForEach-Object { $pipPath = Join-Path $_.FullName "Scripts\pip.exe" if (Test-Path $pipPath) { Write-Host "--- Checking in '$($_.Name)' ---" & $pipPath list | findstr "lifelines" } }

Get-ChildItem -Directory | ForEach-Object {
    $pipPath = Join-Path $_.FullName "Scripts\pip.exe"
    if (Test-Path $pipPath) {
        Write-Host "--- Checking in '$($_.Name)' ---"
        & $pipPath list | findstr "lifelines"
    }
}

Get-ChildItem -Directory | ForEach-Object {
    $pipPath = Join-Path $_.FullName "Scripts\pip.exe"
    if (Test-Path $pipPath) {
        Write-Host "--- Checking in '$($_.Name)' ---"
        & $pipPath list | findstr "lifelines"
    }
}

Inspection results

The post How to check virtual environments appeared first on Customer Experience Management.

Independent variables in simple terms:

• Independent variables: Inputs you control or observe.
• Dependent variable: Output you want to understand or predict.

Example of independent variables in Customer Management (RFM Model):

Suppose you’re analyzing customer behavior to predict churn (whether a customer will stop buying).

• Independent variables:
  • Recency: How recently a customer made a purchase.
  • Frequency: How often they purchase.
  • Monetary: How much they spend.
• Dependent variable:
  • Churn: 1 if the customer churned, 0 if they stayed.

In this case, Recency, Frequency, and Monetary are independent variables used to predict the likelihood of churn. See also Python for business analytics – rfm analysis

Why check for independence among independent variables?

If independent variables are highly correlated with other variables, i.e., it is not truly independent, it can cause multicollinearity, which makes model coefficients unstable, reduces interpretability, and can lead to misleading conclusions.

Multicollinearity

Multicollinearity refers to a statistical phenomenon in which two or more independent variables in a regression model are highly correlated. This makes it difficult to determine the individual effect of each variable on the dependent variable because they essentially carry overlapping information.

Assessment of Multicollinearity

To assess multicollinearity, you can use following methods:

1. Correlation Matrix
   • Check pairwise correlations between independent variables.
   • High correlation (e.g., > 0.8 or < -0.8) may indicate multicollinearity.
2. Variance Inflation Factor (VIF)
   • Measures how much the variance of a regression coefficient is inflated due to multicollinearity.
   • VIF > 5 or 10 is often considered problematic.
3. Tolerance
   • Tolerance = 1 / VIF.
   • Low tolerance values (close to 0) indicate high multicollinearity.
4. Condition Index and Eigenvalues
   • Part of a more advanced diagnostic using matrix decomposition.
   • A condition index > 30 may suggest serious multicollinearity.

How to deal with multicollinearity?

• Remove one of the correlated variables.
• Combine variables (e.g., using PCA or creating an index).
• Regularization techniques like Ridge or Lasso regression.
• Centering variables (subtracting the mean) can help in some cases.

Calculation example

Assume, you have data similar to this sample.

Variable independence testing

import pandas as pd

# Load the data with the specified delimiter
df = pd.read_csv("RFM_analysis_614.csv", delimiter=",")

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Select the independent variables (RFM)
X = df[['recency', 'freq', 'monetary']]

# Add a constant for the VIF calculation (required by the statsmodels function)
X = add_constant(X)

# Create a DataFrame to hold the VIF results
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

# Exclude the constant row from the final output since it's not a true variable
vif_data = vif_data[vif_data.Variable != 'const'].reset_index(drop=True)

print(vif_data)

# Save the VIF results to a CSV file
vif_data.to_csv("vif_results.csv", index=False)

import pandas as pd

# Load the data with the specified delimiter
df = pd.read_csv("RFM_analysis_614.csv", delimiter=",")

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Select the independent variables (RFM)
X = df[['recency', 'freq', 'monetary']]

# Add a constant for the VIF calculation (required by the statsmodels function)
X = add_constant(X)

# Create a DataFrame to hold the VIF results
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

# Exclude the constant row from the final output since it's not a true variable
vif_data = vif_data[vif_data.Variable != 'const'].reset_index(drop=True)

print(vif_data)

# Save the VIF results to a CSV file
vif_data.to_csv("vif_results.csv", index=False)

Finally program prints following results

Based on the Variance Inflation Factor (VIF) calculation, the columns recency, frequency, and monetary are statistically independent of each other and not informationally overlapping.

This means that you can use all three variables together as independent predictors in a statistical model, such as a Cox Proportional Hazards (Cox PH) model, without concern for severe multicollinearity.

Variance Inflation Factor (VIF) Results

The VIF (https://en.wikipedia.org/wiki/Variance_inflation_factor) is a measure of how much the variance of an estimated regression coefficient is increased due to collinearity. A common rule of thumb is that a VIF value less than 5 or sometimes 10 indicates that the correlation between the variables is not high enough to warrant concern.

The calculated VIF values for your RFM variables are very low:

Conclusion on Independence

Since all VIF values are close to 1.0 and well below the 5.0 threshold:

• Independent Variables: You can confidently treat recency, frequency, and monetary as independent variables for your statistical analysis (e.g., in a Cox PH model).
• No Informational Overlap: The variables are providing distinct, non-redundant information to the model. For instance, knowing a customer’s frequency does not allow the model to strongly predict their recency or monetary value.

The post What are independent variables? appeared first on Customer Experience Management.

1. PyCharm

• Developer: JetBrains
• Platforms: Windows, macOS, Linux
• Features: Advanced code analysis, smart code completion, integrated debugger, Git support, virtual environment management, Django support.
• Community (free) and Professional (paid) editions.
• Website: PyCharm

2. Visual Studio Code (VS Code)

• Developer: Microsoft
• Platforms: Windows, macOS, Linux
• Features: Lightweight but powerful; excellent Python extension; integrated terminal; rich plugin ecosystem; Git support; Jupyter notebook integration.
• Website: VS Code

3. Spyder

• Developer: Scientific Python Development Environment Community
• Platforms: Windows, macOS, Linux
• Features: Focused on scientific computing and data science; variable explorer; integrated IPython console; plotting support.
• Website: Spyder

4. Thonny

• Developer: University of Tartu
• Platforms: Windows, macOS, Linux
• Features: Beginner-friendly; simple UI; built-in debugger; good for learning and education.
• Website: Thonny

5. Wing IDE

• Developer: Wingware
• Platforms: Windows, macOS, Linux
• Features: Powerful debugger; code intelligence; remote development support.
• Website: Wing IDE

6. Eric

• Developer: Detlev Offenbach
• Platforms: Windows, macOS, Linux
• Features: Full-featured Python and Ruby IDE; integrated debugger; plugin support.
• Website: Eric Python IDE

7. IDLE

• Developer: Python Software Foundation (bundled with Python)
• Platforms: Windows, macOS, Linux
• Features: Basic, lightweight, good for quick scripts and learning.
• Website: IDLE Documentation

Summary:
For professional development, PyCharm and VS Code are the most popular. For data science, Spyder is widely used. For beginners, Thonny or IDLE are great choices.

The post Top IDEs for Python developers appeared first on Customer Experience Management.

In this tutorial, we’ll show you exactly how to do that. We’ll build a powerful yet simple Python script that automatically scans a webpage, finds all the PDF links, and searches for specific text inside each one. Using popular libraries like Requests, BeautifulSoup, and PyPDF, you’ll learn a practical skill that can save you hours of manual work. Let’s get started!

Python, Web Scraping, PDF, Automation, BeautifulSoup, PyPDF, requests, Data Extraction, Python Projects, Text Search

Scrape a Website

in the script we Find all links on the initial page. Filter for links that end with .pdf. For each PDF link: Download the PDF file into memory. Extract text from every page of the PDF. Search the extracted text for your search_string. And finally Report which PDF files contain the phrase. Scrape a Website

import requests
from bs4 import BeautifulSoup
from pypdf import PdfReader
import io

Scrape a Website
def find_linked_pdfs(url):
    """
    Scans a webpage for PDF links and searches for a string within each PDF.

    Args:
        url: The URL of the webpage to scan.
        search_string: The string to search for inside the PDFs.
    """
    print(f"Scanning {url} for PDF links...")
    try:
        # 1. Get the main page to find all links
        base_url_parts = requests.utils.urlparse(url)
        base_url = f"{base_url_parts.scheme}://{base_url_parts.netloc}"
        
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')

        pdf_links = [a['href'] \
          for a in soup.find_all('a', href=True) \
          if a['href'].endswith('.pdf')]
        
        if not pdf_links:
            print("No PDF links found on the page.")
            return

        print(f"Found {len(pdf_links)} PDF files. Now searching inside them...")

    return pdf_links

import requests
from bs4 import BeautifulSoup
from pypdf import PdfReader
import io

Scrape a Website
def find_linked_pdfs(url):
    """
    Scans a webpage for PDF links and searches for a string within each PDF.

    Args:
        url: The URL of the webpage to scan.
        search_string: The string to search for inside the PDFs.
    """
    print(f"Scanning {url} for PDF links...")
    try:
        # 1. Get the main page to find all links
        base_url_parts = requests.utils.urlparse(url)
        base_url = f"{base_url_parts.scheme}://{base_url_parts.netloc}"
        
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')

        pdf_links = [a['href'] \
          for a in soup.find_all('a', href=True) \
          if a['href'].endswith('.pdf')]
        
        if not pdf_links:
            print("No PDF links found on the page.")
            return

        print(f"Found {len(pdf_links)} PDF files. Now searching inside them...")

    return pdf_links

Handling URLs: It constructs a full, absolute URL for each PDF, as many links on a page can be relative (e.g., /path/to/file.pdf). Scrape a Website. https://pypi.org/project/beautifulsoup4/

In-Memory Processing: Instead of saving each PDF to your disk, it uses io.BytesIO to treat the downloaded content as a file in your computer’s memory. This is faster and cleaner.

Text Extraction: The pypdf library’s PdfReader opens this in-memory file. The script then loops through each page, calls extract_text(), and combines the text from all pages.

Searching and Reporting: Finally, it performs a case-insensitive search on the extracted text and prints the URL of any PDF that contains your search term.

Search Inside PDF
def find_text_in_pdfs(pdf_links, search_string):

        # 2. Loop through each PDF link

        found_in_files = []
        for pdf_path in pdf_links:
            # Construct absolute URL if the link is relative
            if not pdf_path.startswith(('http://', 'https://')):
                pdf_url = f"{base_url}{pdf_path}"
            else:
                pdf_url = pdf_path

            try:
                # 3. Download the PDF content
                pdf_response = requests.get(pdf_url)
                pdf_response.raise_for_status()

                # Use an in-memory buffer to read the PDF
                pdf_file = io.BytesIO(pdf_response.content)
                reader = PdfReader(pdf_file)
                
                # 4. Extract text and search
                full_text = ""
                for page in reader.pages:
                    full_text += page.extract_text() or ""
                
                if search_string.lower() in full_text.lower():
                    print(f"✔️ Found '{search_string}' in: {pdf_url}")
                    found_in_files.append(pdf_url)

            except Exception as e:
                print(f"⚠️ Could not process {pdf_url}. Reason: {e}")
        
        if not found_in_files:
            print(f"\nSearch complete. The string '{search_string}' was not found in any of the PDFs.")

    except requests.exceptions.RequestException as e:
        print(f"An error occurred fetching the main URL: {e}")

Search Inside PDF
def find_text_in_pdfs(pdf_links, search_string):

        # 2. Loop through each PDF link

        found_in_files = []
        for pdf_path in pdf_links:
            # Construct absolute URL if the link is relative
            if not pdf_path.startswith(('http://', 'https://')):
                pdf_url = f"{base_url}{pdf_path}"
            else:
                pdf_url = pdf_path

            try:
                # 3. Download the PDF content
                pdf_response = requests.get(pdf_url)
                pdf_response.raise_for_status()

                # Use an in-memory buffer to read the PDF
                pdf_file = io.BytesIO(pdf_response.content)
                reader = PdfReader(pdf_file)
                
                # 4. Extract text and search
                full_text = ""
                for page in reader.pages:
                    full_text += page.extract_text() or ""
                
                if search_string.lower() in full_text.lower():
                    print(f"✔️ Found '{search_string}' in: {pdf_url}")
                    found_in_files.append(pdf_url)

            except Exception as e:
                print(f"⚠️ Could not process {pdf_url}. Reason: {e}")
        
        if not found_in_files:
            print(f"\nSearch complete. The string '{search_string}' was not found in any of the PDFs.")

    except requests.exceptions.RequestException as e:
        print(f"An error occurred fetching the main URL: {e}")

x

if __name__ == "__main__":
    target_url = "https://www.umcs.pl/pl/plany-zajec,10795.htm"
    search_term = "programming"
    pdf_links = find_linked_pdfs(target_url)
    find_text_in_pdfs(pdf_links, search_string)
    

if __name__ == "__main__":
    target_url = "https://www.umcs.pl/pl/plany-zajec,10795.htm"
    search_term = "programming"
    pdf_links = find_linked_pdfs(target_url)
    find_text_in_pdfs(pdf_links, search_string)
    

Wrapping Up and Next Steps

Congratulations! You’ve successfully built a powerful automation script that bridges the gap between web scraping and document analysis. By combining the strengths of Requests, BeautifulSoup, and PyPDF, you can now programmatically find information that was previously locked away inside PDF files on any website. This not only saves an incredible amount of time but also opens up new possibilities for data collection and analysis. Feel free to adapt the code for your own projects and take your web scraping skills to the next level. Scrape a Website.

The applications for this technique extend far beyond a single use case. Imagine using this script for academic research, automatically scanning university archives for papers mentioning a specific topic. You could adapt it for financial analysis by pulling keywords from dozens of quarterly earnings reports, or for legal work by searching through court filings for a particular case name. Job seekers could even use it to scan company websites for PDF job descriptions that contain key skills.

To perform a statistical analysis of the overall economy, you can leverage a variety of online resources, including government and intergovernmental data portals, as well as academic publications. These sources often provide data in structured formats like CSVs and APIs, but also in less-structured formats like HTML tables and PDFs, which can be parsed using Python libraries like Beautiful Soup and pypdf.

Government and Intergovernmental Data Sources

For raw, official economic data, these are your most reliable sources. They offer a wealth of information on everything from GDP and inflation to employment rates and international trade. Scrape a Website. Search Inside PDF

• Federal Reserve Economic Data (FRED): A fantastic resource from the St. Louis Fed, FRED offers over 800,000 economic time series from more than 100 sources. It’s a goldmine for anyone doing macroeconomic analysis.
• The World Bank Open Data: This portal provides comprehensive global development data, including indicators on economic policy, poverty, gender, and more, making it perfect for cross-country comparisons.
• Data.gov: The home of U.S. government open data, this site aggregates datasets from various federal agencies, including the Bureau of Economic Analysis (BEA) and the Bureau of Labor Statistics (BLS).
• United Nations Statistics Division (UNSD): The UNSD offers a wide array of international statistics, including the UNdata portal which provides free access to over 60 million statistical records from various UN agencies.
• The Bureau of Economic Analysis (BEA): The BEA produces some of the most critical U.S. economic statistics, such as GDP, personal income, and corporate profits.

You can read about Business analyst carrier path here

The core principle remains the same: automate the discovery of information, no matter the format. Search Inside PDF. Happy coding! 🚀

The post How to Scrape a Website and Search Inside PDFs with Python appeared first on Customer Experience Management.

Python’s powerful date and time functions using the datetime and pandas libraries gives you a robust date table ready for Power BI and other business intelligence and analytical tools. Python for analysts most important datetime functions.

Mastering Date and Time Functions in Python for Power BI Date Tables

When working with Power BI, a well-structured Date Table is essential for time intelligence calculations like YTD, QTD, MTD, and custom period comparisons. While Power BI has built-in date table features, using Python to generate a custom date table gives you full control over the structure, granularity, and logic.

In this post, we’ll explore Python’s powerful date and time functions using the datetime and pandas libraries, and show how to create a robust date table ready for Power BI.

Python for analyst – date and time functions – basics

Python provides the datetime module to work with dates and times. Here’s a quick overview:

from datetime import datetime, timedelta, date

# Current date and time
now = datetime.now()
print("Now:", now)

# Just the date
today = date.today()
print("Today:", today)

# Add 7 days
next_week = today + timedelta(days=7)
print("Next week:", next_week)

# Subtract 30 days
last_month = today - timedelta(days=30)
print("30 days ago:", last_month)

These functions are the foundation for generating date ranges and calculating custom columns like fiscal periods or holidays. Python for analysts most important datetime functions.

Creating a Date Range with Pandas

To build a date table, we need a continuous range of dates. pandas.date_range() is perfect for this:

import pandas as pd

# Generate a date range from 2020 to 2030
date_range = pd.date_range(start='2020-01-01', end='2030-12-31', freq='D')
df = pd.DataFrame({'Date': date_range})

This gives us a DataFrame with one row per day — the backbone of our date table.

Enriching the Date Table

Now let’s add useful columns for Power BI:

df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['MonthName'] = df['Date'].dt.strftime('%B')
df['Quarter'] = df['Date'].dt.quarter
df['Day'] = df['Date'].dt.day
df['Weekday'] = df['Date'].dt.weekday + 1  # Monday = 1
df['WeekdayName'] = df['Date'].dt.strftime('%A')
df['IsWeekend'] = df['Weekday'].isin([6, 7])
df['Week'] = df['Date'].dt.isocalendar().week
df['DayOfYear'] = df['Date'].dt.dayofyear

These columns allow for slicing and dicing your data in Power BI by year, month, weekday, and more.

Fiscal Calendar Support

Many businesses use fiscal calendars that don’t align with the calendar year. Here’s how to add a fiscal year starting in July:

df['FiscalYear'] = df['Date'].apply(lambda x: x.year if x.month < 7 else x.year + 1)
df['FiscalMonth'] = df['Date'].apply(lambda x: x.month - 6 if x.month >= 7 else x.month + 6)
df['FiscalQuarter'] = ((df['FiscalMonth'] - 1) // 3) + 1

This logic adjusts the fiscal year, month, and quarter based on a July start. Python for analysts most important datetime functions

5. Flags for Time Intelligence

Power BI benefits from flags that simplify DAX calculations:

today = pd.to_datetime('today').normalize()

df['IsToday'] = df['Date'] == today
df['IsCurrentMonth'] = (df['Date'].dt.month == today.month) & (df['Date'].dt.year == today.year)
df['IsCurrentYear'] = df['Date'].dt.year == today.year

You can also add flags for holidays, fiscal periods, or custom business logic. Python for analysts most important datetime functions

Exporting to CSV for Power BI

Once your date table is ready, export it:

df.to_csv('DateTable.csv', index=False)

You can now import this CSV into Power BI as a static date table.

7. Sample Output

Here’s a preview of what your date table might look like:

Pandas to_datetime() function

Python for analysts most important datetime functions – pandas

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  object        
 3   Dt_last_purchase        13483 non-null  object        

df_cust['Dt_first_purchase'] = pd.to_datetime(df_cust['Dt_first_purchase'],format="yyyy-mm-dd")
df_cust['Dt_last_purchase'] = pd.to_datetime(df_cust['Dt_last_purchase'],format="yyyy-mm-dd")

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  datetime64[ns]
 3   Dt_last_purchase        13483 non-null  datetime64[ns]

Advanced Tips

• Holidays: Use external APIs or CSVs to mark public holidays.
• Week Start: Adjust Weekday to match your locale (e.g., Monday vs. Sunday).
• Time Zones: Use pytz or zoneinfo for timezone-aware datetime handling.
• Dynamic Updates: Automate the script to regenerate the table monthly or yearly.

Certainly! Here’s a concise 400-word post on SQL Date and Time Functions, with examples, tailored for building a Date Table in Power BI:

SQL Date and Time Functions for Power BI Date Tables

When building reports in Power BI, a comprehensive Date Table is essential for enabling time-based calculations like YTD, MTD, and custom period comparisons. While Power BI can auto-generate a date table, using SQL to create one gives you full control over its structure and logic.

Let’s explore key SQL Server date and time functions and how to use them to build a robust date table.

1. Generating a Date Range

To create a date table, you need a continuous range of dates. In SQL Server, you can use a loop or a recursive CTE:

DECLARE @StartDate DATE = '2020-01-01';
DECLARE @EndDate DATE = '2030-12-31';

WITH DateCTE AS (
    SELECT @StartDate AS DateValue
    UNION ALL
    SELECT DATEADD(DAY, 1, DateValue)
    FROM DateCTE
    WHERE DateValue < @EndDate
)
SELECT * INTO DateTable FROM DateCTE
OPTION (MAXRECURSION 32767);

select * from DateTable

Here the example

This creates a table with one row per day between 2020 and 2030.

2. Adding Date Attributes

Once you have the base dates, enrich them with useful columns:

ALTER TABLE DateTable ADD 
    Year INT,
    Month INT,
    MonthName VARCHAR(20),
    Quarter INT,
    Weekday INT,
    WeekdayName VARCHAR(20);

UPDATE DateTable
SET 
    Year = YEAR(DateValue),
    Month = MONTH(DateValue),
    MonthName = DATENAME(MONTH, DateValue),
    Quarter = DATEPART(QUARTER, DateValue),
    Weekday = DATEPART(WEEKDAY, DateValue),
    WeekdayName = DATENAME(WEEKDAY, DateValue);

These columns allow for flexible filtering and grouping in Power BI.

3. Fiscal Calendar and Flags

You can also add fiscal logic and flags:

ALTER TABLE DateTable ADD FiscalYear INT;

UPDATE DateTable
SET FiscalYear = CASE 
    WHEN MONTH(DateValue) >= 7 THEN YEAR(DateValue) + 1
    ELSE YEAR(DateValue)
END;

Add flags like IsWeekend, IsToday, or IsCurrentMonth to simplify DAX expressions.

Conclusion

Python offers a flexible and powerful way to create a custom date table for Power BI. With just a few lines of code, you can generate a rich dataset that supports advanced time intelligence and reporting needs.

Whether you’re working with fiscal calendars, custom flags, or multilingual support, Python gives you the tools to tailor your date table exactly to your business requirements.

SQL’s date and time functions like DATEADD, DATEPART, DATENAME, and YEAR are powerful tools for building a custom date table. Once created, export it to Power BI or use it as a view for dynamic reporting.

Would you like a ready-to-run SQL script for a complete date table?

Find more resources in our course of Advanced programming for business analysts

The post Python for analysts most important datetime functions appeared first on Customer Experience Management.

How AI is changing search behaviors

https://www.nngroup.com/articles/ai-changing-search-behaviors

Semantic search with Elasticsearch – intro

https://www.elastic.co/search-labs/blog/introduction-to-vector-search

Language model implementation

As the first we should initiate a language model and prepare product data for input. Semantic search with Elasticsearch require data indexing via vector transformer.

import pandas as pd
import numpy as np

# function to normalize vectors
def normalize_embedding(embedding):
    norm = np.linalg.norm(embedding)
    return embedding / norm if norm != 0 else embedding

# select products from sql database
def select_products():
    q = """
        SELECT 
         [ProductId]
        ,[ProdIdx]
        ,[ProductName]
        FROM [DB_Products] 
    """
    dfp = read_from_sql_server(q);
    return dfp

# import sentence transformers and initiate model
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
print(model)

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

model = model.to(device)
print(model)

import pandas as pd
import numpy as np

# function to normalize vectors
def normalize_embedding(embedding):
    norm = np.linalg.norm(embedding)
    return embedding / norm if norm != 0 else embedding

# select products from sql database
def select_products():
    q = """
        SELECT 
         [ProductId]
        ,[ProdIdx]
        ,[ProductName]
        FROM [DB_Products] 
    """
    dfp = read_from_sql_server(q);
    return dfp

# import sentence transformers and initiate model
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
print(model)

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

model = model.to(device)
print(model)

Data indexing for semantic search

Now we can index product data into Elasticsearch database (index). We will index product names as vectors and lexically. This allows hybrid search. Semantic search with Elasticsearch.

# initiate Elasticsearch client
from elasticsearch import Elasticsearch, helpers
es = Elasticsearch('http://localhost:9100')

# check es client
from pprint import pprint
pprint(es.info().body)

# ReCreate the index with dense_vector and text mappings
# step 1
es.indices.delete(index="prod_search_hybrid", ignore_unavailable=True)

# step 2 - mappings
es.indices.create(
    index="prod_search_hybrid",
    mappings={
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims": 384,
                "index": True,
                "similarity": "cosine"
            },
            "prodct_name": {
                "type": "text"
            }
        }
    },
)

# list indices and number of documents indexed 
def indices_list():
    indices = es.cat.indices(format='json')
    return [x['index'] for x in indices]
# -------------------
print(indices_list())
# ------------------------------

# Create documents for embedding
documents = []
for i, r in df_docs.iterrows():
    documents.append({        
        'product_name': r['ProductName'][:256].lower(),        
    })

print(f' Created table of {len(documents)} docs')

# Prepare bulk operations
from tqdm import tqdm                                         # for a prograss bar
operations = []
for document in tqdm(documents, total=len(documents)):
    operations.append({'index': {'_index': 'prod_search_hybrid'}})
    operations.append({
        **document,
        'embedding': get_embedding(document['product_name']), # vectors for semantic search
        'product_name': document['product_name']              # the text field for hybrid search
    })

# Bulk insert the data into Elasticsearch
response = es.bulk(operations=operations)
print(f' Records indexed: {len(response["items"])}')

# initiate Elasticsearch client
from elasticsearch import Elasticsearch, helpers
es = Elasticsearch('http://localhost:9100')

# check es client
from pprint import pprint
pprint(es.info().body)

# ReCreate the index with dense_vector and text mappings
# step 1
es.indices.delete(index="prod_search_hybrid", ignore_unavailable=True)

# step 2 - mappings
es.indices.create(
    index="prod_search_hybrid",
    mappings={
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims": 384,
                "index": True,
                "similarity": "cosine"
            },
            "prodct_name": {
                "type": "text"
            }
        }
    },
)

# list indices and number of documents indexed 
def indices_list():
    indices = es.cat.indices(format='json')
    return [x['index'] for x in indices]
# -------------------
print(indices_list())
# ------------------------------

# Create documents for embedding
documents = []
for i, r in df_docs.iterrows():
    documents.append({        
        'product_name': r['ProductName'][:256].lower(),        
    })

print(f' Created table of {len(documents)} docs')

# Prepare bulk operations
from tqdm import tqdm                                         # for a prograss bar
operations = []
for document in tqdm(documents, total=len(documents)):
    operations.append({'index': {'_index': 'prod_search_hybrid'}})
    operations.append({
        **document,
        'embedding': get_embedding(document['product_name']), # vectors for semantic search
        'product_name': document['product_name']              # the text field for hybrid search
    })

# Bulk insert the data into Elasticsearch
response = es.bulk(operations=operations)
print(f' Records indexed: {len(response["items"])}')

Hybrid search

For hybrid search we combine match and knn search inside a bool query. The _name field return what what part of the query has returned results. This allow to build hybride scoring. As you can see we vectorize query_h to query_vector using the same procedure get_embeding as we were using during indexing. Semantic search with Elasticsearch.

query_h = "anti-explosion device"

# Print the query vector for debugging
# query_vector = get_embedding(query_h)
# print("Query Vector:", query_vector)

response = es.search(
    index='prod_search_hybrid',
    body={
        "query": {
            "bool": {
                "should": [
                    {
                        "match": {
                            "product_name": {
                                "query": query_h,
                                "_name": "text_match"
                            }
                        }
                    },
                    {
                        "knn": {
                            "field": "embedding",
                            "query_vector":  query_vector,
                            "k": 10,
                            "num_candidates": 100,
                            "_name": "semantic_search"
                        }
                    }
                ]
            }
        }
        ,
        'size': 30
    }
)

query_h = "anti-explosion device"

# Print the query vector for debugging
# query_vector = get_embedding(query_h)
# print("Query Vector:", query_vector)

response = es.search(
    index='prod_search_hybrid',
    body={
        "query": {
            "bool": {
                "should": [
                    {
                        "match": {
                            "product_name": {
                                "query": query_h,
                                "_name": "text_match"
                            }
                        }
                    },
                    {
                        "knn": {
                            "field": "embedding",
                            "query_vector":  query_vector,
                            "k": 10,
                            "num_candidates": 100,
                            "_name": "semantic_search"
                        }
                    }
                ]
            }
        }
        ,
        'size': 30
    }
)

After that we can extract product names and scoring and build a list according to our intention. In here we separate products according to field _name and then build list of top 10 lexical match and semantic similarity. More about similarity measures you can read in post Measuring product similarity

# Extract product names, scores, and sources
products = [
    (hit["_source"]["product_name"], hit["_score"], hit.get("matched_queries", []))
    for hit in response["hits"]["hits"]
]

# Separate products into text_match and semantic_search groups
text_match_products = [product for product in products if "text_match" in product[2]]
semantic_search_products = [product for product in products if "semantic_search" in product[2]]

# Sort each group by score in descending order
sorted_text_match_products = sorted(text_match_products, key=lambda x: x[1], reverse=True)[:10]
sorted_semantic_search_products = sorted(semantic_search_products, key=lambda x: x[1], reverse=True)[:10]

# Print top 10 text_match products
print("\nTop 10 Text Match Products:")
for product in sorted_text_match_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

# Print top 10 semantic_search products
print("\nTop 10 Semantic Search Products:")
for product in sorted_semantic_search_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

# Extract product names, scores, and sources
products = [
    (hit["_source"]["product_name"], hit["_score"], hit.get("matched_queries", []))
    for hit in response["hits"]["hits"]
]

# Separate products into text_match and semantic_search groups
text_match_products = [product for product in products if "text_match" in product[2]]
semantic_search_products = [product for product in products if "semantic_search" in product[2]]

# Sort each group by score in descending order
sorted_text_match_products = sorted(text_match_products, key=lambda x: x[1], reverse=True)[:10]
sorted_semantic_search_products = sorted(semantic_search_products, key=lambda x: x[1], reverse=True)[:10]

# Print top 10 text_match products
print("\nTop 10 Text Match Products:")
for product in sorted_text_match_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

# Print top 10 semantic_search products
print("\nTop 10 Semantic Search Products:")
for product in sorted_semantic_search_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

Full-text search, also known as lexical search, is a technique for fast, efficient searching through text fields in documents. Documents and search queries are transformed to enable returning relevant results instead of simply exact term matches. Fields of type text are analyzed and indexed for full-text search.

You can combine full-text search with semantic search using vectors to build modern hybrid search applications. While vector search may require additional GPU resources, the full-text component remains cost-effective by leveraging existing CPU infrastructure.

Another example of vector indexing and sementic search you can find here: https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb

Vector search setup and performing hybrid search

https://www.elastic.co/search-labs/blog/vector-search-set-up-elasticsearch

https://www.elastic.co/search-labs/blog/hybrid-search-elasticsearch

Filtering. Semantic search with Elasticsearch

Filter context is mostly used for filtering structured data. For example, use filter context to answer questions like:

• Does this timestamp fall into the range 2015 to 2016?
• Is the status field set to “published”?

Filter context is in effect whenever a query clause is passed to a filter parameter, such as the filter or must_not parameters in a bool query. Learn more about filter context in the Elasticsearch docs.

Keyword Filtering

This is an example of adding a keyword filter to the query. The example retrieves the top books that are similar to “javascript books” based on their title vectors, and also Addison-Wesley as publisher. Semantic search with Elasticsearch.

response = client.search(
    index="book_index",
    knn={
        "field": "title_vector",
        "query_vector": model.encode("javascript books"),
        "k": 10,
        "num_candidates": 100,
        "filter": {"term": {"publisher.keyword": "addison-wesley"}},
    },
)

pprint(response)

Query and filter context

Relevance scores

By default, Elasticsearch sorts matching search results by relevance score, which measures how well each document matches a query. The relevance score is a positive floating point number, returned in the _score metadata field of the search API. The higher the _score, the more relevant the document. While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a query or filter context.

Query context

In the query context, a query clause answers the question How well does this document match this query clause? Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the _score metadata field. Query context is in effect whenever a query clause is passed to a query parameter, such as the query parameter in the search API. Semantic search with Elasticsearch

Filter context

A filter answers the binary question “Does this document match this query clause?”. The answer is simply “yes” or “no”. Filtering has several benefits:

1. Simple binary logic: In a filter context, a query clause determines document matches based on a yes/no criterion, without score calculation.
2. Performance: Because they don’t compute relevance scores, filters execute faster than queries.
3. Caching: Elasticsearch automatically caches frequently used filters, speeding up subsequent search performance.
4. Resource efficiency: Filters consume less CPU resources compared to full-text queries.
5. Query combination: Filters can be combined with scored queries to refine result sets efficiently.

Filters are particularly effective for querying structured data and implementing “must have” criteria in complex searches.

Structured data refers to information that is highly organized and formatted in a predefined manner. In the context of Elasticsearch, this typically includes:

• Numeric fields (integers, floating-point numbers)
• Dates and timestamps
• Boolean values
• Keyword fields (exact match strings)
• Geo-points and geo-shapes

Unlike full-text fields, structured data has a consistent, predictable format, making it ideal for precise filtering operations. Semantic search with Elasticsearch.

Common filter applications include:

• Date range checks: for example is the timestamp field between 2015 and 2016
• Specific field value checks: for example is the status field equal to “published” or is the author field equal to “John Doe”

Filter context applies when a query clause is passed to a filter parameter, such as:

• filter or must_not parameters in bool queries
• filter parameter in constant_score queries
• filter aggregations

Filters optimize query performance and efficiency, especially for structured data queries and when combined with full-text searches.

GET /_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title":   "Search"        }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [
        { "term":  { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}

Read more: Product Search and Product classification for E-commerce

Reciprocal rank fusion

Reciprocal rank fusion (RRF) is a method for combining multiple result sets with different relevance indicators into a single result set. RRF requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results. Semantic search with Elasticsearch.

RRF uses the following formula to determine the score for ranking each document:

score = 0.0
for q in queries:
    if d in result(q):
        score += 1.0 / ( k + rank( result(q), d ) )
return score

# where
# k is a ranking constant
# q is a query in the set of queries
# d is a document in the result set of q
# result(q) is the result set of q
# rank( result(q), d ) is d's rank within the result(q) starting from 1

You can use RRF as part of a search to combine and rank documents using separate sets of top documents (result sets) from a combination of child retrievers using an RRF retriever. A minimum of two child retrievers is required for ranking.

We rank the documents based on the RRF formula with a rank_window_size of 5 truncating the bottom 2 docs in our RRF result set with a size of 3. We end with _id: 3 as _rank: 1, _id: 2 as _rank: 2, and _id: 4 as _rank: 3. This ranking matches the result set from the original RRF search as expected.

In this example, we execute the knn and standard retrievers independently of each other. Then we use the rrf retriever to combine the results.

1. First, we execute the kNN search specified by the knn retriever to get its global top 50 results.
2. Second, we execute the query specified by the standard retriever to get its global top 50 results.
3. Then, on a coordinating node, we combine the kNN search top documents with the query top documents and rank them based on the RRF formula using parameters from the rrf retriever to get the combined top documents using the default size of 10.

Note that if k from a knn search is larger than rank_window_size, the results are truncated to rank_window_size. If k is smaller than rank_window_size, the results are k size.

GET example-index/_search
{
    "retriever": {
        "rrf": {
            "retrievers": [
                {
                    "standard": {
                        "query": {
                            "term": {
                                "text": "shoes"
                            }
                        }
                    }
                },
                {
                    "knn": {
                        "field": "vector",
                        "query_vector": [1.25, 2, 3.5],
                        "k": 50,
                        "num_candidates": 100
                    }
                }
            ],
            "rank_window_size": 50,
            "rank_constant": 20
        }
    }
}

The post Semantic Search with Elasticsearch appeared first on Customer Experience Management.

Key terminology of survival analysis models

• Event: The outcome of interest, such as death, disease occurrence, customer churn, or equipment failure.
• Time: The duration from a defined starting point (e.g., start of treatment or customer acquisition) to the occurrence of the event or the end of observation (censoring).
• Censoring: When the event of interest is not observed for some individuals during the study period, making their exact survival time unknown.
• Survival Function S(t): The survival function S(t) is defined as the probability that a subject survives (i.e., does not experience the event) beyond time t.
• Hazard Function h(t): The hazard function h(t) represents the instantaneous rate at which the event occurs at time t, given that the subject has survived up to that time.
• Linking the Survival and Hazard Functions: The survival and hazard functions are mathematically related. Knowing one allows you to derive the other, reflecting the duality between retention (survival) and churn (event occurrence).

Linking the Survival and Hazard Functions

The survival and hazard functions are mathematically related. Knowing one allows you to derive the other, reflecting the duality between retention (survival) and churn (event occurrence).

In summary, the cumulative hazard rate of subject i at time t can also be defined as the negative logarithm of the survival function at time t.

Stratification

When we have a variable influence in some moment of time, we can divide the period of observation to two periods, when the time before the variable application and the the time post. It constitutes a new starting point of the new treatment application to the patient. The shift of the survival function could be observed.

Kaplan-Meier Estimator

Kaplan-Meier Estimator is a non-parametric model for estimating the survival function, often used as a first step in survival analysis. It demonstrate the probability of survival a certain time in function of time. The maximum value is 1 and the K-M curve goes approximately to zero when time increases.

Cox Proportional Hazards Model

Cox Proportional Hazards Model is a semi-parametric model that incorporate the influence of some predictor variables to the hazard rate or survival model.  Each subject has an observed survival time 0-t and an event variable xE that shows whether the event has occurred. It is also observed in time 0-t tat variables x1 .. n have had an influence on event xE occurrence. The Cox model incorporate the influence of x1 .. xn factors to estimation the time of survival or hazard rate as n explanatory variables. This make the model can also predict a time of survival.

Disclaimer: the post is inspired by: Benjamin Lee, A Comparison Study of Parametric and Machine Learning Survival Analysis Models to Predict Customer Churn in the Edtech Sector, Vienna, 27th January, 2025
Benjamin Lee, Peter Filzmoser, link: here

Dataset for Cox Proportional Hazards Model

If we have dataset like here

The Kaplan-Meier estimator is the statistical tool used to estimate a true survival function from available data and can be considered the ’best’ estimator of survival probability when no parametric structure is assumed. This estimator is a non-parametric estimator that only requires the time-to-event (or time-to-censoring) t , and the event status e for every subject. With this information, the survival function estimator S(t) is given by:

where e_j is the time at which at least one event e occurred, and n_j is the total number of subjects who have been censored or have not had the event yet at time t_j .

One of the main drawbacks of the Kaplan-Meier model is that it is not able to take any subject covariates into account – what means, it does not provide explanatory power in terms of different variables within a group.

Understanding the Cox Proportional Hazards Model

When we want to understand how factors like age, treatment type, or blood pressure affect survival time, the Cox Proportional Hazards model is a go-to tool. Instead of trying to pin down the exact risk at every single moment, this model cleverly separates the hazard rate into two distinct parts.

The core idea is to model how specific factors (or covariates) have a multiplicative effect on an underlying hazard rate. The formula for the model looks like this:

h(t,X)=h₀​(t)⋅exp(βX)

Let’s break that down:

• h_0​(t) is the Baseline Hazard. Think of this as the underlying risk of the event for a “standard” individual (where all covariate values are zero) over time. It’s the part of the model that changes with time (t), but it’s “non-parametric,” meaning we don’t assume its shape. We let the data speak for itself.
• exp(βX) is the Covariate Effect. This is the parametric part of the model and it’s where your specific factors come in.
  • X is the set of your covariates (e.g., age, treatment type, or blood pressure).
  • β are the coefficients, similar to those in a linear regression, that the model estimates.
  • This component tells us how much an individual’s unique characteristics increase or decrease their risk compared to the baseline. Applying the exponential function, exp(), ensures this multiplier is always positive, as negative risk doesn’t make sense.

Notice that only the baseline hazard depends on time. The covariate effect, exp(βX), is constant over time. This separation is the key to the model’s power and leads directly to its most important assumption.

In the Cox proportional hazards model, a covariate is a predictor variable that you include in the model to explain or predict the risk (hazard) of a specific event occurring over time, such as death, failure, or relapse. These covariates can be continuous (e.g., age, blood pressure) or categorical (e.g., treatment group, gender).

What the program calculates:

• The model estimates the effect of each covariate on the hazard function—the instantaneous risk of the event happening at a certain time.
• It computes coefficients (often denoted as ββ) for each covariate, quantifying how the risk changes with a one-unit increase in that variable, assuming other covariates remain constant.
• These coefficients translate into hazard ratios (exponentiated coefficients), which tell how much the hazard (risk) is multiplied by when the covariate changes by one unit.
• The baseline hazard function h0(t)h0(t) represents the hazard if all covariates were zero; the model then scales this baseline hazard depending on the values of the covariates for each individual.
• Overall, the Cox model evaluates how your covariates modify the risk of the event over survival time, assuming that these effects multiply the baseline hazard proportionally and do not change with time (proportional hazards assumption).

Formally, the hazard function is modeled as

where x1,x2,…,xpx1,x2,…,xp are your covariates and β1,β2,…,βpβ1,β2,…,βp are their estimated effects.

Summary:

• Your covariate is any variable in your data that may affect the timing of the event.
• The Cox model calculates how these covariates affect the hazard or instantaneous risk of the event happening, summarized through hazard ratios.
• This helps you understand which factors increase or decrease risk while accounting for the survival time and censoring in your dataset.

This explanation aligns with standard survival analysis literature and is consistent with how lifelines or R coxph functions use covariates in the Cox model.

The “Proportional” in Proportional Hazards

The model’s name comes from its core assumption: the effect of the covariates is constant over time. In other words, the hazard ratio between any two individuals remains proportional throughout the entire timeline.

Let’s make this concrete. Imagine we have a single covariate, x – age, and two subjects, ‘a’ and ‘b’. Their hazard rates are:

• Subject a: h(t,x_a​)=h_0​(t)⋅exp(βx_a​)
• Subject b: h(t,x_b​)=h₀​(t)⋅exp(βx_b​)

If we look at the ratio of their hazards, the baseline hazard h₀​(t) cancels out:

As you can see, time (t) has vanished from the right side of the equation. This means that if subject ‘a’ has double the risk of subject ‘b’ on day 1, they will also have double the risk on day 100 or day 500. The risk difference is because of age difference. Their relative risk is constant, or proportional. This powerful assumption is a direct consequence of the model’s structure.

If one has a dataset which violates the proportional hazards assumption, it causes reduction in predictive power of the model. The hazard as an output from the model is a useful tool in assessing the general magnitude of a covariates’ effect on the time of survival. One can interpret the hazard ratio as the weighted average of true hazard ratios over the time period. Therefore, one must not strictly conform to the proportional hazards assumption, but always check if dataset is appropriate to this assumption.

You can check if your dataset meets the proportional hazards assumption of the Cox model using both visual plots and formal statistical tests.

Visual Inspection (Graphical Methods)

Visual checks are often the first and most intuitive step.

Log-Log Survival Plots

This is a classic method for categorical covariates (like “treatment group” vs. “control group”).

• How it works: You plot the logarithm of time, log(t), on the x-axis against a special transformation of the survival probability, -log(-log(S(t))), on the y-axis for each group.
• What to look for: If the proportional hazards assumption holds, the resulting curves for each group should be roughly parallel and not cross. If the lines cross or move closer or further apart in a systematic way, the assumption may be violated.

Schoenfeld Residuals Plots

This is the most common and powerful visual method, and it works for both continuous and categorical covariates.

• How it works: For each event that occurs, a residual is calculated that represents the difference between the observed covariate value and the expected covariate value for the individual who had the event. You then plot these residuals against time.
• What to look for: If the assumption holds, you should see a random scatter of points around a horizontal line at zero. If you see any clear pattern or trend (e.g., a line with a positive or negative slope), it suggests the effect of the covariate changes over time, violating the assumption.

Formal Statistical Tests

A formal test gives you a p-value to help you decide if any violations you see are statistically significant.

• How it works: The most common test is based on the Schoenfeld residuals. It formally tests whether the slope of a line fitted to the Schoenfeld residuals-vs-time plot is significantly different from zero. This is often done using a function like cox.zph() in R or the check_assumptions() method in Python’s lifelines library.
• How to interpret the results:
  • Null Hypothesis (H_0): The effect of the covariate is constant over time (the proportional hazards assumption holds).
  • A low p-value (e.g., < 0.05) suggests you should reject the null hypothesis. This is evidence that the assumption is violated for that specific covariate.
  • A high p-value means you fail to reject the null hypothesis, so it’s reasonable to assume the proportional hazards assumption is met.

Example of check_assumptions(data) in Python

The dataset includes:

• Subject ID
• Survival time in days
• Event status (1 = event occurred, 0 = censored)
• Age group (1 – 4)

To use check_assumptions() from lifelines, you need to fit a Cox Proportional Hazards model, which requires at least one covariate (a variable that might affect survival, ex age group).

from lifelines import CoxPHFitter

# Fit the Cox model
cph = CoxPHFitter()
cph.fit(data, duration_col="Survival_in_days", event_col="Status_bool")

# Check proportional hazards assumption
cph.check_assumptions(data)
# or
cph.check_assumptions(data, p_value_threshold=0.05)

What to Do If the Assumption is Violated

If you find that a key variable violates the assumption, you have options:

1. Stratification: You can stratify your model by the problematic covariate. This allows the baseline hazard function to be different for each level of that variable, resolving the violation. For example, if “gender” violates the assumption, you can stratify by it.
2. Use Time-Dependent Covariates: You can modify the model to include an interaction term between the covariate and time. This explicitly models how the covariate’s effect changes over time.
3. Choose a Different Model: If the violation is severe, a parametric model (like a Weibull or Log-Logistic model) that has a built-in shape for the hazard function might be a more appropriate choice for your data.

Parametric Models

When the underlying probability distribution of the dataset is known, one can use parametric models to model the survival function of the dataset. Once the underlying model is specified either in terms of the survival times or the logarithm of survival times, the model can be fitted and estimated using the maximum likelihood estimator.

Of course. Parametric models in survival analysis assume that a subject’s survival time follows a specific, known statistical distribution (like the exponential or Weibull distribution). Unlike non-parametric models (e.g., Kaplan-Meier) which don’t make assumptions about the data’s distribution, parametric models are defined by a set of parameters that dictate the shape of the survival and hazard functions.

Understanding them means understanding the hazard function each model assumes. The hazard function describes the instantaneous risk of an event occurring at a specific time, given that it hasn’t occurred yet.

Exponential Model

This is the simplest parametric model. It assumes the hazard rate is constant over time.

• Core Idea: The risk of the event happening is the same every single day. If a component hasn’t failed by day 10, its risk of failing on day 11 is the same as it was on day 2.
• Hazard Function: h(t)=λ (a constant)
• How to Understand It: This model is defined by a single parameter, λ (lambda), the hazard rate. It’s best suited for events that don’t have a “memory” or aging process, such as the failure rate of certain electronic components or the occurrence of random external events.

Weibull Model

The Weibull model is a more flexible and widely used model because it does not assume a constant hazard rate.

• Core Idea: The risk of the event can increase, decrease, or remain constant over time. This makes it much more adaptable to real-world scenarios.
• Hazard Function: h(t)=λk(λt)k−1
• How to Understand It: The model is defined by two main parameters:
  • λ (scale parameter): Stretches or compresses the curve.
  • k (shape parameter): This is the key. It dictates the nature of the hazard.
    • If k>1, the hazard increases over time (e.g., aging, where the risk of failure grows).
    • If k<1, the hazard decreases over time (e.g., post-surgery recovery, where the initial risk is high but drops).
    • If k=1, the Weibull model simplifies to the Exponential model with a constant hazard.

Log-Normal and Log-Logistic Models

These models are useful for situations where the hazard rate is not monotonic (i.e., it doesn’t just go up or down).

• Log-Normal Model
  • Core Idea: Assumes that the logarithm of the survival time follows a normal (bell-shaped) distribution.
  • Hazard Function: The hazard rate first increases to a peak and then decreases.
  • How to Understand It: Think of situations where failure is most likely after a certain “wear-in” period, but if the subject survives past that peak, the immediate risk then declines. It’s often used in engineering for component fatigue.
• Log-Logistic Model
  • Core Idea: Similar to the Log-Normal model, it also assumes the logarithm of survival time follows a specific distribution (the logistic distribution).
  • Hazard Function: The hazard rate can be hump-shaped (increasing then decreasing) or monotonically decreasing, depending on its parameters. It’s more flexible than the Log-Normal model.
  • How to Understand It: This model is popular in medical research, especially when studying diseases like cancer where the risk of mortality might peak some time after diagnosis and then fall. It’s also notable because its survival function has a simple, explicit formula, which makes interpreting odds easier.

Accelerated Failure Time (AFT) models

In contrast to the Cox model, AFT models assume that covariates are proportional with respect to survival time.

Metrics for Survival Analysis

Log-Rank Test

The log-rank test is used to compare two or more survival functions with each other. In this sense, it is analogous to the t-test or Pearson’s chi-squared test for survival analysis. Like those tests, the log-rank test tests the null hypothesis H0 that there is no difference between the survival functions being compared in the probability of an event e occurring at any time t.

The Log-Rank Test is a statistical test used in survival analysis to compare the survival distributions of two or more groups. It’s particularly useful for determining if there are significant differences in the time it takes for an event (like death, failure, or relapse) to occur between groups.

Example Calculation

Let’s consider a clinical trial comparing the survival times of patients using two different cancer treatments, Drug A and Drug B. Here’s a simplified dataset:

Steps to Calculate the Log-Rank Test:

1. Combine the Data: List all unique time points from both groups.
2. Calculate the Number at Risk: For each time point, determine how many patients are still at risk in each group.
3. Calculate the Expected Events: For each time point, calculate the expected number of events (deaths) for each group.
4. Compute the Test Statistic: Use the observed and expected events to calculate the test statistic.

Detailed Calculation:

1. Combine the Data:
   • Time points: 2, 3, 4, 6, 7, 8
2. Number at Risk:
   • At time 2: Group A: 2, Group B: 3
   • At time 3: Group A: 1, Group B: 3
   • At time 4: Group A: 1, Group B: 2
   • At time 6: Group A: 1, Group B: 2
   • At time 7: Group A: 0, Group B: 2
   • At time 8: Group A: 0, Group B: 1
3. Expected Events:
   • At time 2: Expected events for Group A = (2/5) * 1 = 0.4, Group B = (3/5) * 1 = 0.6
   • At time 4: Expected events for Group A = (1/3) * 1 = 0.33, Group B = (2/3) * 1 = 0.67
   • At time 6: Expected events for Group A = (1/3) * 1 = 0.33, Group B = (2/3) * 1 = 0.67
   • At time 8: Expected events for Group A = 0, Group B = 1
4. Test Statistic:
   • Sum the observed and expected events for each group.
   • Calculate the test statistic using the formula:
   $ \chi^2 = \sum \frac{(Oi – Ei)^2}{E_i} $ Where ( Oi ) is the observed number of events and ( Ei ) is the expected number of events. For our example, the test statistic would be calculated based on the observed and expected events at each time point.

If the test statistic is greater than the critical value from the chi-square distribution table (with 1 degree of freedom), we reject the null hypothesis and conclude that there is a significant difference between the survival distributions of the two groups [1] [2].

Example of Survival Analysis Models in Python

The code demonstrate a plot of survival and hazard functions.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# --- 1. Create a Sample Dataset ---
data_dict = {
    'time': [6, 7, 10, 15, 18, 22, 25, 30, 32, 38, 40, 45],
    'event': [1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1]  # 1 = event, 0 = censored
}
data = pd.DataFrame(data_dict)
data = data.sort_values(by='time').reset_index(drop=True)

# --- 2. Calculate Kaplan-Meier Survival Function ---
km_data = data.copy()
unique_times = sorted(km_data['time'].unique())
survival_prob = 1.0
results = []

for t in unique_times:
    at_risk = (km_data['time'] >= t).sum()
    events = km_data[(km_data['time'] == t) & (km_data['event'] == 1)].shape[0]
    
    if at_risk > 0:
        survival_prob *= (1 - events / at_risk)
    
    results.append({'time': t, 'survival': survival_prob})

km_results = pd.DataFrame(results)

# --- 3. Plot the Survival Function ---
plt.figure(figsize=(10, 6))
plt.step(km_results['time'], km_results['survival'], where='post', label='Kaplan-Meier Estimate')
plt.scatter(data[data['event'] == 0]['time'], km_results.loc[data[data['event']==0].index, 'survival'],
            marker='+', color='red', s=100, label='Censored')
plt.title('Survival Function (Kaplan-Meier Estimate)')
plt.xlabel('Time (Months)')
plt.ylabel('Survival Probability')
plt.ylim(0, 1.05)
plt.xlim(0, max(data['time']) + 5)
plt.grid(True)
plt.legend()
plt.savefig('manual_survival_function_plot.png')
plt.close()

# --- 4. Calculate Nelson-Aalen Cumulative Hazard ---
na_data = data.copy()
hazard = 0.0
na_results_list = []

for t in unique_times:
    at_risk = (na_data['time'] >= t).sum()
    events = na_data[(na_data['time'] == t) & (na_data['event'] == 1)].shape[0]

    if at_risk > 0:
        hazard += events / at_risk
    
    na_results_list.append({'time': t, 'cumulative_hazard': hazard})

na_results = pd.DataFrame(na_results_list)

# --- 5. Plot the Cumulative Hazard Function ---
plt.figure(figsize=(10, 6))
plt.step(na_results['time'], na_results['cumulative_hazard'], where='post', label='Nelson-Aalen Estimate')
plt.title('Cumulative Hazard Function (Nelson-Aalen Estimate)')
plt.xlabel('Time (Months)')
plt.ylabel('Cumulative Hazard')
plt.xlim(0, max(data['time']) + 5)
plt.grid(True)
plt.legend()
plt.savefig('manual_cumulative_hazard_plot.png')
plt.close()

[1]: DATAtab [2]: Real Statistics Using Excel

References

[1] Log-Rank Test: A Beginner’s Guide – DATAtab

[2] Log-Rank Test – Real Statistics Using Excel

The post Survival Analysis Models appeared first on Customer Experience Management.

Course Structure:

Week 1-2: Introduction and Setup

• Lecture: Overview of SQL and Python in business analytics.
• Lab: Setting up the environment (SQL databases, Python IDEs).
• Project 1: Data Import and Cleaning
  • SQL: Importing data from various sources, cleaning and preprocessing.
  • Python: Using pandas for data cleaning and manipulation.

Week 3-4: Data Exploration and Visualization

• Lecture: Techniques for data exploration and visualization.
• Lab: SQL queries for data exploration, Python libraries for visualization (matplotlib, seaborn).
• Project 2: Exploratory Data Analysis (EDA)
  • SQL: Writing complex queries to explore data.
  • Python: Visualizing data trends and patterns.

Week 5-6: Statistical Analysis

• Lecture: Statistical methods for business analytics.
• Lab: SQL functions for statistical analysis, Python libraries (numpy, scipy).
• Project 3: Statistical Analysis of Sales Data
  • SQL: Calculating statistical measures (mean, median, standard deviation).
  • Python: Performing hypothesis testing and regression analysis.

Week 7-8: Predictive Modeling

• Lecture: Introduction to predictive modeling techniques.
• Lab: SQL for data preparation, Python for model building (scikit-learn).
• Project 4: Predictive Sales Forecasting
  • SQL: Preparing data for modeling.
  • Python: Building and evaluating predictive models.

Week 9-10: Time Series Analysis

• Lecture: Time series analysis and forecasting.
• Lab: SQL for time series data manipulation, Python libraries (statsmodels).
• Project 5: Time Series Forecasting
  • SQL: Extracting and transforming time series data.
  • Python: Building time series models and forecasting.

Week 11-12: Machine Learning Integration

• Lecture: Integrating machine learning with SQL databases.
• Lab: SQL for data storage and retrieval, Python for machine learning (TensorFlow, Keras).
• Project 6: Customer Segmentation
  • SQL: Storing and retrieving data for machine learning.
  • Python: Building and deploying machine learning models.

Week 13-14: Advanced Topics and Final Project

• Lecture: Advanced topics in SQL and Python (optimization, big data).
• Lab: SQL performance tuning, Python for big data (PySpark).
• Project 7: Comprehensive Business Analytics Project
  • SQL: Optimizing queries for large datasets.
  • Python: Analyzing and visualizing large datasets.

Assessment:

• Projects: Each mini project will be assessed based on accuracy, efficiency, and creativity.
• Final Project: A comprehensive project integrating all learned skills.

Resources:

• Books: “SQL for Data Analytics” by Upom Malik, Matt Goldwasser, and Benjamin Johnston; “Python for Data Analysis” by Wes McKinney.
• Online Tutorials: SQLZoo, DataCamp, Coursera.

Advanced Programming in SQL and Python

Advanced Programming in Data Analysis

https://datascience.umcs.pl

The post Advanced Programming in SQL and Python – the course for business analysts appeared first on Customer Experience Management.

Row date we can get via this query

select [Id], [Name], html
 FROM [DBPromoHeader]
where Html is not null and Html like '%href%'

We got data like this

We can use string_split to convert html field to url.

select [Id], [Name], html, replace(replace(value,'href="','https://mietwood.com/'),'"','') Promo_url
 FROM [DBPromoHeader]
 cross apply string_split(Html,' ')
 where Html is not null and Html like '%href%' and value like 'href%'
 order by [Id]

After that we got following result

Happy coding.

The post Python for analyst – string split appeared first on Customer Experience Management.

Wizualizacja danych jest kluczowym elementem analizy danych, ponieważ umożliwia szybkie zrozumienie wzorców, zależności i trendów ukrytych w danych. W Pythonie dwie najbardziej popularne biblioteki do tworzenia wizualizacji to Matplotlib i Seaborn. Każda z nich ma swoje zalety i zastosowania, dlatego w tym artykule porównamy je, aby pomóc Ci wybrać odpowiednie narzędzie dla Twoich potrzeb.

Autorem opracowania jest Natalia Kustra, Analityka gospodarcza I rok, II stopień, 2025

1. Wprowadzenie do Matplotlib i Seaborn

Matplotlib

Matplotlib to najbardziej wszechstronna i podstawowa biblioteka do tworzenia wykresów w Pythonie. Oferuje ogromną elastyczność, umożliwiając tworzenie niemal każdego rodzaju wykresu, od prostych linii po skomplikowane wykresy 3D. Jednak ta elastyczność często wymaga pisania większej ilości kodu.

Przykład prostego wykresu liniowego w Matplotlib:

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]
y = [10, 12, 8, 15, 10]

plt.plot(x, y, label=’Dane’)
plt.title(‘Wykres liniowy’)
plt.xlabel(‘X’)
plt.ylabel(‘Y’)
plt.legend()
plt.show()

Seaborn

Seaborn to biblioteka zbudowana na podstawie Matplotlib, która koncentruje się na uproszczeniu procesu tworzenia wizualizacji. Jest szczególnie użyteczna dla analizy danych statystycznych i oferuje bardziej estetyczne domyślne style wykresów.

Przykład wykresu punktowego w Seaborn:

import seaborn as sns
import matplotlib.pyplot as plt

# Przykładowe dane
import pandas as pd
import numpy as np
data = pd.DataFrame({
    ‘X’: np.random.rand(50),
    ‘Y’: np.random.rand(50)
})

sns.scatterplot(x=’X’, y=’Y’, data=data)
plt.title(‘Wykres punktowy’)
plt.show()

2. Porównanie kluczowych cech

Elastyczność

• Matplotlib:
  • Umożliwia pełną kontrolę nad każdym aspektem wykresu.
  • Sprawdzi się w przypadku niestandardowych i zaawansowanych wizualizacji.
  • Kod bywa dłuższy i bardziej skomplikowany.
• Seaborn:
  • Jest zoptymalizowany do tworzenia standardowych wykresów statystycznych, takich jak wykresy pudełkowe, histogramy, czy wykresy gęstości.
  • Idealny do szybkiej eksploracji danych.
  • Mniejsza elastyczność w porównaniu z Matplotlib.

Łatwość użycia

• Matplotlib:
  • Wymaga bardziej szczegółowej konfiguracji wykresów.
  • Dla początkujących może być trudniejsza w użyciu.
• Seaborn:
  • Intuicyjne API pozwala na tworzenie skomplikowanych wizualizacji za pomocą kilku linii kodu.
  • Świetnie integruje się z biblioteką Pandas.

Wygląd wykresów

• Matplotlib:
  • Domyślne style wykresów są podstawowe i wymagają dostosowania.
  • Wymaga ręcznego dostrajania, aby uzyskać estetyczne efekty.
• Seaborn:
  • Domyślnie oferuje bardziej estetyczne i nowoczesne style.
  • Obsługuje palety kolorów, które są przydatne w analizie danych statystycznych.

Wydajność

• Matplotlib:
  • Może być nieco wolniejszy przy tworzeniu dużych wykresów, zwłaszcza w przypadku skomplikowanych animacji.
• Seaborn:
  • Działa szybciej dla prostych wizualizacji, ale bazuje na Matplotlib, więc w dużej mierze dzieli jej ograniczenia wydajnościowe.

3. Przykłady praktyczne

Wykres słupkowy: Matplotlib vs. Seaborn

Matplotlib:

categories = [‘A’, ‘B’, ‘C’, ‘D’]
values = [5, 7, 3, 8]

plt.bar(categories, values, color=’skyblue’)
plt.title(‘Wykres słupkowy w Matplotlib’)
plt.xlabel(‘Kategorie’)
plt.ylabel(‘Wartości’)
plt.show()

Seaborn:

data = pd.DataFrame({‘Kategorie’: [‘A’, ‘B’, ‘C’, ‘D’], ‘Wartości’: [5, 7, 3, 8]})

sns.barplot(x=’Kategorie’, y=’Wartości’, data=data, palette=’Blues’)
plt.title(‘Wykres słupkowy w Seaborn’)
plt.show()

Wykres pudełkowy:

Matplotlib:

values = [7, 8, 5, 6, 9, 10, 6, 7, 8]

plt.boxplot(values)
plt.title(‘Wykres pudełkowy w Matplotlib’)
plt.show()

Seaborn:

sns.boxplot(data=values, color=’lightblue’)
plt.title(‘Wykres pudełkowy w Seaborn’)
plt.show()

4. Kiedy wybrać Matplotlib, a kiedy Seaborn?

• Wybierz Matplotlib, jeśli:
  • Potrzebujesz pełnej kontroli nad wyglądem wykresów.
  • Tworzysz niestandardowe wizualizacje lub animacje.
  • Chcesz korzystać z funkcji 3D.
• Wybierz Seaborn, jeśli:
  • Chcesz szybko tworzyć estetyczne wykresy z danymi w Pandas.
  • Analizujesz dane statystyczne.
  • Potrzebujesz domyślnie dobrze wyglądających wykresów.

5. Podsumowanie

Matplotlib i Seaborn to potężne narzędzia do wizualizacji danych, ale każde z nich ma swoje unikalne zastosowania. Matplotlib zapewnia elastyczność i kontrolę, podczas gdy Seaborn pozwala szybko tworzyć estetyczne wizualizacje. Wybór między nimi zależy od Twoich potrzeb i doświadczenia.

Jeśli dopiero zaczynasz, warto zacząć od Seaborn, aby zaoszczędzić czas i skupić się na analizie danych. W miarę zdobywania doświadczenia Matplotlib pozwoli Ci na tworzenie bardziej zaawansowanych wykresów.

The post Matplotlib vs Seaborn appeared first on Customer Experience Management.