Python – Customer Experience Management https://mietwood.com Customer Experience Can Be Managed Wed, 31 Dec 2025 11:44:55 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://mietwood.com/wp-content/uploads/2022/09/cropped-Fav7-32x32.png Python – Customer Experience Management https://mietwood.com 32 32 Product Grouping in Noisy E-commerce Datasets https://mietwood.com/product-grouping-in-noisy-e-commerce-datasets Tue, 30 Dec 2025 09:34:33 +0000 https://mietwood.com/?p=3438 Hybrid Approaches to Product Grouping in Noisy E-commerce Datasets: A Comparative Analysis of Set-Theoretic vs. Vector Space Models. In scientific literature, this problem is formally known as Short Text Clustering (STC) or Product Entity Resolution. The product similarity measures can be read from here: Measuring product similarity – 5 important secrets of python programming. Machine...

The post Product Grouping in Noisy E-commerce Datasets appeared first on Customer Experience Management.

]]> Hybrid Approaches to Product Grouping in Noisy E-commerce Datasets: A Comparative Analysis of Set-Theoretic vs. Vector Space Models. In scientific literature, this problem is formally known as Short Text Clustering (STC) or Product Entity Resolution. The product similarity measures can be read from here: Measuring product similarity – 5 important secrets of python programming. Machine learning of product clustering you can find here: Hierarchical Agglomerative Clustering for Product Grouping.

The “Marketplace Deduplication” Problem

Context: Large marketplaces (Amazon, eBay, Alibaba, or a niche aggregator) allow thousands of third-party sellers to upload their own product feeds. Product Grouping for E-commerce Datasets indeed.

The Problem:

  • Seller A uploads: “Apple iPhone 13, 128GB, Midnight”
  • Seller B uploads: “iPhone 13 128 GB Black Unlocked”
  • Seller C uploads: “Smartfon Apple iPhone 13 128GB (MLPF3PM/A)”
Why it’s hard – The Scientific Challenge
  • Missing Identifiers: Sellers often omit EAN/UPC codes to avoid price comparisons.
  • Attribute Noise: “Midnight” vs. “Black” (synonyms).
  • Goal: You must cluster these into a single catalog entry (The “Golden Record”) to show the user one product page with a list of 3 sellers, rather than 3 separate search results.
  • ** Metric:** False Positives are costly here (grouping an iPhone 13 Pro with a regular iPhone 13 causes returns).

The “Omnichannel Customer Stitching” Problem (Single Customer View)

Context: A retailer sells through a Website, a Mobile App, and Physical Stores. They want to know if the person browsing the app is the same person buying in the store.

The Problem:

  • Record A (Online): email: j.smith@gmail.com, cookie_id: xyz123, behavior: viewed running shoes
  • Record B (In-Store POS): card_hash: ****-1234, loyalty_id: 998877, name: John Smith
  • Record C (Customer Support): phone: +48 500..., name: Johnny Smith, complaint: "Shoes size 42 too small"

Why it’s hard:

Why it’s hard
  • Disjoint Attributes: Record A has no phone number; Record C has no email. You need “transitive linking” (A links to B, B links to C $\rightarrow$ A links to C).
  • Privacy/Hashing: You are often matching hashed values or partial PII (Personally Identifiable Information).
  • Goal: Create a Customer 360 profile to send a targeted email: “Hi John, sorry the size 42 didn’t fit. Here is a discount for size 43.”

The “Competitor Price Monitoring” Problem

Context: An e-commerce store wants to automatically adjust their prices to be $1 cheaper than their biggest competitor.

The Problem:

  • Your Product: “Samsung Galaxy S20 FE 5G Cloud Navy”
  • Competitor Site: “Samsung S20 Fan Edition (Navy) – 5G Compatible”

Why it’s hard:

Why it’s hard – The Scientific Challenge:
  • Adversarial Data: Competitors intentionally slightly alter names or use unique internal SKUs to prevent scraping and matching.
  • Asymmetry: You have your full database (structured), but the competitor data is scraped (unstructured, noisy HTML).
  • Goal: Map competitor SKUs to your SKUs with high precision. If you map to the wrong product (e.g., a cheaper “Lite” version), your dynamic pricing algorithm will lower your price too much and you lose money.

Abstract

  • Problem: E-commerce catalogs suffer from redundancy (same product, different sizes/variants).
  • Gap: Manual grouping is impossible; Deep Learning is overkill/imprecise for strict SKU grouping.
  • Method: We compare Jaccard (Set) vs. TF-IDF (Vector) and propose a normalization pipeline.
  • Result: Our method achieved X% accuracy with Y% reduction in computational time.

Set-Theoretic Approaches for Product Grouping (Jaccard)

  • Concept: Treats text as a “Bag of Words” (BoW) without weights.
  • Key Papers/Concepts:
    • Cohen et al. (2003) often discuss string metrics for entity matching.
    • Shingling / MinHash: In large datasets, calculating Jaccard for all pairs is $O(N^2)$. Literature focuses on Locality Sensitive Hashing (LSH) (MinHash) to approximate Jaccard similarity efficiently.
    • Pros in Literature: High interpretability, excellent for “near-duplicate” detection.
    • Cons: Fails when synonyms are used (e.g., “pants” vs “trousers”) or when word importance varies.

Vector Space Models (TF-IDF)

  • Concept: Maps text to a high-dimensional Euclidean space.
  • Key Papers/Concepts:
    • Salton et al. (1975) (The foundational VSM paper).
    • Character n-grams: Papers often cite that for noisy user-generated content (UGC), character n-grams outperform word tokens because they handle misspellings morphologically.
    • Pros: Handles “rare words” (like model numbers) better due to IDF (Inverse Document Frequency) weighting.

The State-of-the-Art (Deep Learning / Embeddings)

  • If you publish, reviewers will ask: “Why not BERT?”
  • SBERT (Sentence-BERT): Current SOTA uses transformer models to generate dense vector embeddings.
  • Your Counter-Argument: Deep learning is computationally expensive and “black-box”. For industrial product grouping where exact feature matching (like “Samsung” + “Galaxy”) is critical, classical methods (TF-IDF/Jaccard) often offer better precision and control than semantic embeddings which might group “iPhone 12” with “Samsung S20” because they are both “phones”.

II. Related Work

  • Mention LSH (Locality Sensitive Hashing) for Jaccard.
  • Mention DBSCAN and Agglomerative Clustering as standard algorithms.
  • Cite limitations of BERT in high-precision SKU matching.
  • Product Grouping for E-commerce

III. Methodology (The Core)

Group definition in dataset
  • Define SKU vs Product Group (Parent-Child relationship).
  • The challenge: “Noise” in titles (e.g., 500ml, XL, Pack of 2).
  1. Preprocessing – The Normalization Filter for Product Grouping for E-commerce:
    • Define your Regex rules mathematically.
    • $T_{clean} = f(T_{raw})$ where $f$ removes tokens $t \in \{Dimensions, Colors, Stopwords\}$.
  2. Representation:
    • Approach A (Jaccard): $J(A,B) = \frac{|A \cap B|}{|A \cup B|}$
    • Approach B (TF-IDF): Cosine Similarity $\cos(\theta) = \frac{A \cdot B}{||A|| ||B||}$
  3. Clustering Algorithm:
    • Explain why you chose Connected Components (Graph theory) for Jaccard or Hierarchical Clustering for TF-IDF.

IV. Experiments (Product Grouping for E-commerce)

  • Dataset: Your dataset of “several thousand products”.
  • Metrics: You must measure quality.
    • Precision: Are elements in the cluster actually the same product?
    • Recall: Did we find all sizes of that product?
    • F1-Score: Harmonic mean of the two.

V. Results & Discussion

  • Hypothesis: Jaccard works better for clean data; TF-IDF works better for noisy data (typos).
  • Observation: Jaccard is faster but TF-IDF + N-grams is more robust.

Novel Algorithm:

  1. Stage 1 (Blocking): Use Jaccard on tokens to quickly find “Candidate Pairs” (fast, rough filter).
  2. Stage 2 (Refinement): Use TF-IDF with Character N-grams on the candidates to calculate a precise similarity score (handles typos).
  3. Stage 3 (Decision): Hard threshold (e.g., >0.85).

Relevant Search Terms & Papers

Entity Resolution (ER) is the problem of identifying which records in a database refer to the same real-world entity. An exhaustive ER process involves computing the similarities between pairs of records, which can be very expensive for large datasets. Various blocking techniques can be used to enhance the performance of ER by dividing the records into blocks in multiple ways and only comparing records within the same block.

However, most blocking techniques process blocks separately and do not exploit the results of other blocks. In this paper (https://dl.acm.org/doi/pdf/10.1145/1559845.1559870), authors propose an iterative blocking framework where the ER results of blocks are reflected to subsequently processed blocks. Blocks are now iteratively processed until no block contains any more matching records.

Compared to simple blocking, iterative blocking may achieve higher accuracy because reflecting the ER results of blocks to other blocks may generate additional record matches. Iterative blocking may also be more efficient because processing a block now saves the processing time for other blocks.

  • “Short Text Clustering for E-commerce”
  • “Product Entity Resolution with Noise”
  • “Comparison of Jaccard and Cosine Similarity in Text Mining”
  • “Blocking techniques for Entity Resolution” (This is crucial for scaling to thousands/millions of products). Product Grouping for E-commerce.

Specific types of papers to look for:

  1. Koporec et al.: Papers on combining Jaccard with other metrics. https://www.sciencedirect.com/science/article/pii/S1364815225002981
  2. Ganesan et al.: Research on “abstractive summarization” or short-text clustering in retail.

FAQ: Product Grouping in Noisy E-Commerce Datasets

1. What does “noisy data” mean in e-commerce?

Noisy data refers to inconsistencies, errors, or irrelevant information in product listings—such as misspellings, incomplete descriptions, or duplicate entries—that make grouping products challenging.

2. Why is product grouping important?

Grouping similar products improves search accuracy, recommendation quality, and overall user experience. It also helps businesses manage inventory and pricing strategies more effectively

3. What are common challenges in product grouping?
  • Variations in product names and descriptions
  • Missing or incorrect attributes
  • Multiple languages or regional differences
  • Inconsistent categorization by sellers
4. Which techniques are used to handle noisy datasets?
  • Text normalization (removing special characters, standardizing case)
  • Tokenization and similarity measures (e.g., cosine similarity, Jaccard index)
  • Machine learning models for clustering and classification
  • Attribute-based matching using structured data
5. Can AI improve product grouping accuracy?


Yes. AI models like BERT or domain-specific embeddings can capture semantic meaning in product descriptions, making grouping more accurate even with noisy data.

6. How do I start implementing product grouping?


Begin with data cleaning and normalization, then apply similarity-based clustering or train a supervised model if labeled data is available.

The post Product Grouping in Noisy E-commerce Datasets appeared first on Customer Experience Management.

]]> How to check virtual environments https://mietwood.com/virtual-environment Thu, 25 Sep 2025 19:19:18 +0000 https://mietwood.com/?p=3321 You can check virtual environments using a simple for loop in your terminal. This command finds all subdirectories in a specified parent folder, assumes each is a virtual environment, and then uses that environment’s pip to check for lifelines. How it works: ## For Windows 🪟 You can use a similar loop in either Command...

The post How to check virtual environments appeared first on Customer Experience Management.

]]> You can check virtual environments using a simple for loop in your terminal. This command finds all subdirectories in a specified parent folder, assumes each is a virtual environment, and then uses that environment’s pip to check for lifelines.

  1. Navigate to the folder that contains all your virtual environments.
  2. Run the following command: Bash for venv in */ ; do if [ -f "${venv}bin/pip" ]; then echo "--- Checking in '${venv%?}' ---" ${venv}bin/pip list | grep 'lifelines' fi done

How it works:

  • It loops through each subdirectory (e.g., my_project_venv/).
  • It checks if a pip executable exists inside the bin folder to confirm it’s likely a venv.
  • It then runs the pip list command from within that specific environment and uses grep to filter for the line containing “lifelines”.
  • If lifelines is installed, it will print the package name and its version. If not, it will print nothing for that environment.

## For Windows 🪟

You can use a similar loop in either Command Prompt (CMD) or PowerShell.

In PowerShell:

  1. Open PowerShell and navigate to the folder containing your virtual environments.
  2. Run the following command: PowerShe ll Get-ChildItem -Directory | ForEach-Object { $pipPath = Join-Path $_.FullName "Scripts\pip.exe" if (Test-Path $pipPath) { Write-Host "--- Checking in '$($_.Name)' ---" & $pipPath list | findstr "lifelines" } }
Get-ChildItem -Directory | ForEach-Object {
    $pipPath = Join-Path $_.FullName "Scripts\pip.exe"
    if (Test-Path $pipPath) {
        Write-Host "--- Checking in '$($_.Name)' ---"
        & $pipPath list | findstr "lifelines"
    }
}

Inspection results

The post How to check virtual environments appeared first on Customer Experience Management.

]]> What are independent variables? https://mietwood.com/independent-variables Thu, 25 Sep 2025 18:44:48 +0000 https://mietwood.com/?p=3316 Independent variables, also called predictors, features, or explanatory variables, are the variables in a statistical or machine learning model that are used to explain or predict changes in another variable — the dependent variable, also called the outcome or target. Independent variables in simple terms: Example of independent variables in Customer Management (RFM Model): Suppose...

The post What are independent variables? appeared first on Customer Experience Management.

]]> Independent variables, also called predictors, features, or explanatory variables, are the variables in a statistical or machine learning model that are used to explain or predict changes in another variable — the dependent variable, also called the outcome or target.

Independent variables in simple terms:

  • Independent variables: Inputs you control or observe.
  • Dependent variable: Output you want to understand or predict.

Example of independent variables in Customer Management (RFM Model):

Suppose you’re analyzing customer behavior to predict churn (whether a customer will stop buying).

  • Independent variables:
    • Recency: How recently a customer made a purchase.
    • Frequency: How often they purchase.
    • Monetary: How much they spend.
  • Dependent variable:
    • Churn: 1 if the customer churned, 0 if they stayed.

In this case, Recency, Frequency, and Monetary are independent variables used to predict the likelihood of churn. See also Python for business analytics – rfm analysis

Why check for independence among independent variables?

If independent variables are highly correlated with other variables, i.e., it is not truly independent, it can cause multicollinearity, which makes model coefficients unstable, reduces interpretability, and can lead to misleading conclusions.

Multicollinearity

Multicollinearity refers to a statistical phenomenon in which two or more independent variables in a regression model are highly correlated. This makes it difficult to determine the individual effect of each variable on the dependent variable because they essentially carry overlapping information.

Assessment of Multicollinearity

To assess multicollinearity, you can use following methods:

  1. Correlation Matrix
    • Check pairwise correlations between independent variables.
    • High correlation (e.g., > 0.8 or < -0.8) may indicate multicollinearity.
  2. Variance Inflation Factor (VIF)
    • Measures how much the variance of a regression coefficient is inflated due to multicollinearity.
    • VIF > 5 or 10 is often considered problematic.
  3. Tolerance
    • Tolerance = 1 / VIF.
    • Low tolerance values (close to 0) indicate high multicollinearity.
  4. Condition Index and Eigenvalues
    • Part of a more advanced diagnostic using matrix decomposition.
    • A condition index > 30 may suggest serious multicollinearity.

How to deal with multicollinearity?

  • Remove one of the correlated variables.
  • Combine variables (e.g., using PCA or creating an index).
  • Regularization techniques like Ridge or Lasso regression.
  • Centering variables (subtracting the mean) can help in some cases.

Calculation example

Assume, you have data similar to this sample.

RFM data sample – for testing independent variables

Variable independence testing

import pandas as pd

# Load the data with the specified delimiter
df = pd.read_csv("RFM_analysis_614.csv", delimiter=",")

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Select the independent variables (RFM)
X = df[['recency', 'freq', 'monetary']]

# Add a constant for the VIF calculation (required by the statsmodels function)
X = add_constant(X)

# Create a DataFrame to hold the VIF results
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

# Exclude the constant row from the final output since it's not a true variable
vif_data = vif_data[vif_data.Variable != 'const'].reset_index(drop=True)

print(vif_data)

# Save the VIF results to a CSV file
vif_data.to_csv("vif_results.csv", index=False)

Finally program prints following results

Based on the Variance Inflation Factor (VIF) calculation, the columns recency, frequency, and monetary are statistically independent of each other and not informationally overlapping.

This means that you can use all three variables together as independent predictors in a statistical model, such as a Cox Proportional Hazards (Cox PH) model, without concern for severe multicollinearity.

Variance Inflation Factor (VIF) Results

The VIF (https://en.wikipedia.org/wiki/Variance_inflation_factor) is a measure of how much the variance of an estimated regression coefficient is increased due to collinearity. A common rule of thumb is that a VIF value less than 5 or sometimes 10 indicates that the correlation between the variables is not high enough to warrant concern.

The calculated VIF values for your RFM variables are very low:

VariableVIF
recency1.437
freq1.488
monetary1.065

Conclusion on Independence

Since all VIF values are close to 1.0 and well below the 5.0 threshold:

  • Independent Variables: You can confidently treat recency, frequency, and monetary as independent variables for your statistical analysis (e.g., in a Cox PH model).
  • No Informational Overlap: The variables are providing distinct, non-redundant information to the model. For instance, knowing a customer’s frequency does not allow the model to strongly predict their recency or monetary value.

The post What are independent variables? appeared first on Customer Experience Management.

]]> Top IDEs for Python developers https://mietwood.com/ides-for-python-developers Thu, 25 Sep 2025 09:11:31 +0000 https://mietwood.com/?p=3312 Here a top graphical IDEs (Integrated Development Environments) for Python developers as of 2024: 1. PyCharm 2. Visual Studio Code (VS Code) 3. Spyder 4. Thonny 5. Wing IDE 6. Eric 7. IDLE Summary:For professional development, PyCharm and VS Code are the most popular. For data science, Spyder is widely used. For beginners, Thonny or IDLE are great choices.

The post Top IDEs for Python developers appeared first on Customer Experience Management.

]]> Here a top graphical IDEs (Integrated Development Environments) for Python developers as of 2024:

1. PyCharm

  • Developer: JetBrains
  • Platforms: Windows, macOS, Linux
  • Features: Advanced code analysis, smart code completion, integrated debugger, Git support, virtual environment management, Django support.
  • Community (free) and Professional (paid) editions.
  • Website: PyCharm

2. Visual Studio Code (VS Code)

  • Developer: Microsoft
  • Platforms: Windows, macOS, Linux
  • Features: Lightweight but powerful; excellent Python extension; integrated terminal; rich plugin ecosystem; Git support; Jupyter notebook integration.
  • Website: VS Code

3. Spyder

  • Developer: Scientific Python Development Environment Community
  • Platforms: Windows, macOS, Linux
  • Features: Focused on scientific computing and data science; variable explorer; integrated IPython console; plotting support.
  • Website: Spyder

4. Thonny

  • Developer: University of Tartu
  • Platforms: Windows, macOS, Linux
  • Features: Beginner-friendly; simple UI; built-in debugger; good for learning and education.
  • Website: Thonny

5. Wing IDE

  • Developer: Wingware
  • Platforms: Windows, macOS, Linux
  • Features: Powerful debugger; code intelligence; remote development support.
  • Website: Wing IDE

6. Eric

  • Developer: Detlev Offenbach
  • Platforms: Windows, macOS, Linux
  • Features: Full-featured Python and Ruby IDE; integrated debugger; plugin support.
  • Website: Eric Python IDE

7. IDLE

  • Developer: Python Software Foundation (bundled with Python)
  • Platforms: Windows, macOS, Linux
  • Features: Basic, lightweight, good for quick scripts and learning.
  • Website: IDLE Documentation

Summary:
For professional development, PyCharm and VS Code are the most popular. For data science, Spyder is widely used. For beginners, Thonny or IDLE are great choices.

The post Top IDEs for Python developers appeared first on Customer Experience Management.

]]> How to Scrape a Website and Search Inside PDFs with Python https://mietwood.com/scrape-a-website-and-search-inside-pdfs-with-python Sat, 30 Aug 2025 09:13:30 +0000 https://mietwood.com/?p=3292 Ever found yourself on a webpage with dozens of PDF links, needing to find a specific piece of information buried in one of them? 😩 We will teach you how to scrape a website and search inside PDFs with Python. Manually downloading and searching each file is tedious, time-consuming, and prone to errors. What if...

The post How to Scrape a Website and Search Inside PDFs with Python appeared first on Customer Experience Management.

]]> Ever found yourself on a webpage with dozens of PDF links, needing to find a specific piece of information buried in one of them? 😩 We will teach you how to scrape a website and search inside PDFs with Python. Manually downloading and searching each file is tedious, time-consuming, and prone to errors. What if you could automate the entire process with just a few lines of code?

In this tutorial, we’ll show you exactly how to do that. We’ll build a powerful yet simple Python script that automatically scans a webpage, finds all the PDF links, and searches for specific text inside each one. Using popular libraries like Requests, BeautifulSoup, and PyPDF, you’ll learn a practical skill that can save you hours of manual work. Let’s get started!

Python, Web Scraping, PDF, Automation, BeautifulSoup, PyPDF, requests, Data Extraction, Python Projects, Text Search

Scrape a Website

in the script we Find all links on the initial page. Filter for links that end with .pdf. For each PDF link: Download the PDF file into memory. Extract text from every page of the PDF. Search the extracted text for your search_string. And finally Report which PDF files contain the phrase. Scrape a Website

import requests
from bs4 import BeautifulSoup
from pypdf import PdfReader
import io

Scrape a Website
def find_linked_pdfs(url):
    """
    Scans a webpage for PDF links and searches for a string within each PDF.

    Args:
        url: The URL of the webpage to scan.
        search_string: The string to search for inside the PDFs.
    """
    print(f"Scanning {url} for PDF links...")
    try:
        # 1. Get the main page to find all links
        base_url_parts = requests.utils.urlparse(url)
        base_url = f"{base_url_parts.scheme}://{base_url_parts.netloc}"
        
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')

        pdf_links = [a['href'] \
          for a in soup.find_all('a', href=True) \
          if a['href'].endswith('.pdf')]
        
        if not pdf_links:
            print("No PDF links found on the page.")
            return

        print(f"Found {len(pdf_links)} PDF files. Now searching inside them...")

    return pdf_links

Handling URLs: It constructs a full, absolute URL for each PDF, as many links on a page can be relative (e.g., /path/to/file.pdf). Scrape a Website. https://pypi.org/project/beautifulsoup4/

In-Memory Processing: Instead of saving each PDF to your disk, it uses io.BytesIO to treat the downloaded content as a file in your computer’s memory. This is faster and cleaner.

Text Extraction: The pypdf library’s PdfReader opens this in-memory file. The script then loops through each page, calls extract_text(), and combines the text from all pages.

Searching and Reporting: Finally, it performs a case-insensitive search on the extracted text and prints the URL of any PDF that contains your search term.

Search Inside PDF
def find_text_in_pdfs(pdf_links, search_string):

        # 2. Loop through each PDF link

        found_in_files = []
        for pdf_path in pdf_links:
            # Construct absolute URL if the link is relative
            if not pdf_path.startswith(('http://', 'https://')):
                pdf_url = f"{base_url}{pdf_path}"
            else:
                pdf_url = pdf_path

            try:
                # 3. Download the PDF content
                pdf_response = requests.get(pdf_url)
                pdf_response.raise_for_status()

                # Use an in-memory buffer to read the PDF
                pdf_file = io.BytesIO(pdf_response.content)
                reader = PdfReader(pdf_file)
                
                # 4. Extract text and search
                full_text = ""
                for page in reader.pages:
                    full_text += page.extract_text() or ""
                
                if search_string.lower() in full_text.lower():
                    print(f"✔️ Found '{search_string}' in: {pdf_url}")
                    found_in_files.append(pdf_url)

            except Exception as e:
                print(f"⚠️ Could not process {pdf_url}. Reason: {e}")
        
        if not found_in_files:
            print(f"\nSearch complete. The string '{search_string}' was not found in any of the PDFs.")

    except requests.exceptions.RequestException as e:
        print(f"An error occurred fetching the main URL: {e}")

x

if __name__ == "__main__":
    target_url = "https://www.umcs.pl/pl/plany-zajec,10795.htm"
    search_term = "programming"
    pdf_links = find_linked_pdfs(target_url)
    find_text_in_pdfs(pdf_links, search_string)
Scrape a Website and Search Inside PDFs with Python
Scrape a Website and Search Inside PDFs with Python

Wrapping Up and Next Steps

Congratulations! You’ve successfully built a powerful automation script that bridges the gap between web scraping and document analysis. By combining the strengths of Requests, BeautifulSoup, and PyPDF, you can now programmatically find information that was previously locked away inside PDF files on any website. This not only saves an incredible amount of time but also opens up new possibilities for data collection and analysis. Feel free to adapt the code for your own projects and take your web scraping skills to the next level. Scrape a Website.

The applications for this technique extend far beyond a single use case. Imagine using this script for academic research, automatically scanning university archives for papers mentioning a specific topic. You could adapt it for financial analysis by pulling keywords from dozens of quarterly earnings reports, or for legal work by searching through court filings for a particular case name. Job seekers could even use it to scan company websites for PDF job descriptions that contain key skills.

To perform a statistical analysis of the overall economy, you can leverage a variety of online resources, including government and intergovernmental data portals, as well as academic publications. These sources often provide data in structured formats like CSVs and APIs, but also in less-structured formats like HTML tables and PDFs, which can be parsed using Python libraries like Beautiful Soup and pypdf.

Government and Intergovernmental Data Sources

For raw, official economic data, these are your most reliable sources. They offer a wealth of information on everything from GDP and inflation to employment rates and international trade. Scrape a Website. Search Inside PDF

  • Federal Reserve Economic Data (FRED): A fantastic resource from the St. Louis Fed, FRED offers over 800,000 economic time series from more than 100 sources. It’s a goldmine for anyone doing macroeconomic analysis.
  • The World Bank Open Data: This portal provides comprehensive global development data, including indicators on economic policy, poverty, gender, and more, making it perfect for cross-country comparisons.
  • Data.gov: The home of U.S. government open data, this site aggregates datasets from various federal agencies, including the Bureau of Economic Analysis (BEA) and the Bureau of Labor Statistics (BLS).
  • United Nations Statistics Division (UNSD): The UNSD offers a wide array of international statistics, including the UNdata portal which provides free access to over 60 million statistical records from various UN agencies.
  • The Bureau of Economic Analysis (BEA): The BEA produces some of the most critical U.S. economic statistics, such as GDP, personal income, and corporate profits.

You can read about Business analyst carrier path here

The core principle remains the same: automate the discovery of information, no matter the format. Search Inside PDF. Happy coding! 🚀

The post How to Scrape a Website and Search Inside PDFs with Python appeared first on Customer Experience Management.

]]> Semantic Search https://mietwood.com/semantic-search Wed, 20 Aug 2025 11:33:03 +0000 https://mietwood.com/?p=3279 Since the advent of ChatGPT in November 2022, there is not a single day goes by without hearing or reading about vector or semantic search. It’s everywhere and so prevalent that we often get the impression this is a new cutting-edge technology. Vector vs lexical search An easy way to introduce vector search is by...

The post Semantic Search appeared first on Customer Experience Management.

]]> Since the advent of ChatGPT in November 2022, there is not a single day goes by without hearing or reading about vector or semantic search. It’s everywhere and so prevalent that we often get the impression this is a new cutting-edge technology.

Vector vs lexical search

An easy way to introduce vector search is by comparing it to the more conventional lexical search that you’re probably used to. Vector search, also commonly known as semantic search, and lexical search work very differently.

Lexical search is the kind of search that we’ve all been using for years. To summarize it very briefly, it doesn’t try to understand the real meaning of what is indexed and queried, instead, it makes a big effort to lexically match the literals of the words or variants of them like stemming words, or synonyms, etc.. That makes what the user types in a query with all the literals that have been previously indexed into the database. The similarity is replaced by ranking algorithm, such as TF-IDF.

Documents are tokenized and analyzed. Then, the resulting terms are indexed in an inverted index, which simply maps the analyzed terms to the documents containing them. Searching for “yellow texas roses” will match all documents with varying scores.

Semantic search – the whole purpose of semantic search is to index data in such a way that it can be searched based on the meaning it represents.

What’s the difference between semantic search and lexical search?

Lexical search doesn’t try to understand the real meaning of what is indexed and queried- it matches the literals of the words or their variants. In contrast, vector search indexes data in a way that allows it to be searched based on the meaning it represents.

Read more about vector similarity https://www.elastic.co/search-labs/blog/introduction-to-vector-search

Semantic search with elasticsearch

Load model. See details here and here. 

from elasticsearch import Elasticsearch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-base")
VECTOR_DIMENSION = model.get_sentence_embedding_dimension()
print(f"Model loaded successfully.VECTOR_DIMENSION {VECTOR_DIMENSION}")

Start elastic server

from elasticsearch import Elasticsearch
ES_HOST = "http://localhost:9000"
es = Elasticsearch(hosts=[ES_HOST])
print(f"Connection successful: {es.info().body['cluster_name']}")

Index product data to elastic database

# Fetch products from database
def select_products():
    q = """
        SELECT *
        FROM [DB_Products] 
        where [BrandName] in ('PCE','De Walt')
        """
    dfp = read_from_sql_server(q, odbc_conect='DSN=SQLxx')
    return dfp

# >-------------------------------------------------
df_docs = select_products()
print(df_docs.info())
# -------------------------------------------------
# Index product names into elastic
index_mapping = {
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims":  VECTOR_DIMENSION,
                "index": True,
                "similarity": "cosine"
                },

            "product_name": { 
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256 
                    }
                  }
                },
            "product_name_org": {
                "type": "keyword",
                "ignore_above": 256
                },
            "prodidx": {
                "type": "keyword",
                "ignore_above": 256
                },
            "img_url": {
                "type": "keyword",
                "ignore_above": 512 
                },
            }
        }

# --- 4. Create the Index -------------------------------------------------

INDEX_NAME = 'prod_names_for_search_hybrid'

import time
es.options(ignore_status=[400, 404]).indices.delete(index=INDEX_NAME, ignore_unavailable=True)
time.sleep(3) # Give a moment for index deletion to propagate

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, mappings=index_mapping)
    print("Index created.")
else:
    print("Index already exists.")
# -----------------------------------------------------------------------

# Indexing product data ------------------------------------------------- 
product_names = df_docs['ProductName'].apply(lambda x: x.lower()[:256]).to_list()

i = 1
for name in product_names:
    print(f"Indexing:{i} {name}")
    i += 1
    
    vector = model.encode(f"passage: {name}").tolist()
    doc = {
        "Product_name": name,
        "ProductNameVector": vector
    }
    
    es.index(index=INDEX_NAME, document=doc, refresh=True)

# ---------------------------------------------------------------------

You can check your index using diagnostic functions. Semantic search.

# --- Diagnostic functions  ---------------------------------------
def indices_list():
    return [index['index'] for index in es.cat.indices(format='json')]   

def count_records(index_name):
    return es.count(index=index_name)['count']    
# -----------------------------------------------------------------

# Get indices and record count ------------------------------------
for i in indices_list():
    print(i, count_records(i))

# Get mapings
mapping = es.indices.get_mapping(index=INDEX_NAME)
fields = mapping[INDEX_NAME]['mappings']['properties']
for field, details in fields.items():
    print(f"Model {INDEX_NAME}, fields {field}: {details['type']}")
#------------------------------------------------------------------

Semantic search

Now when you have index you can search. You can read more in my post Semantic Search with Elasticsearch

# Semantic Search --- 

query_text = "yellow texas rose"
query_vector = model.encode(f"query: {query_text}").tolist()
knn_query = {
    "field": "embedding", #"ProductNameVector",
    "query_vector": query_vector,
    "k": 10,
    "num_candidates": 50
}
response = es.search(index=INDEX_NAME, knn=knn_query, source=["product_name"])
for hit in response['hits']['hits']:
    print(f"  - Product: {hit['_source']['product_name']} (Score: {hit['_score']:.4f})")

You can perform also lexical search or finnali hybrid search.

def search_score_script(query_h):
    response = es.search(
        index=INDEX_NAME,
        body={
            "_source": ['product_name','prodidx','img_url'],
            "query": {
                "function_score": {
                    "query": {
                        "bool": {
                            "should": [
                                {
                                    "match": {
                                        "product_name": {
                                            "query": query_h,
                                            'operator': 'and',
                                            "_name": "text_match"
                                        }
                                    }
                                },
                                {
                                    "knn": {
                                        "field": "embedding",
                                        "query_vector": model.encode(f"query: {query_text}").tolist(),
                                        "k": 30,
                                        "num_candidates": 300,
                                        "_name": "semantic_search"
                                    }
                                }
                            ]
                        }
                    },                 
                }
            },
            "size": 100
        }
    )
    return response

response = search_score_script(query_h)   
products = [
         {
             "product_name": hit["_source"]["product_name"],
             "score": hit["_score"],
             "matched_queries": hit.get("matched_queries", []),
             "prodidx": hit["_source"]["prodidx"],
         }
         for hit in response["hits"]["hits"]
     ]

print([(p["product_name"],p["score"] for p in products])

Sentence Transformers (e.g., MiniLM, BERT variants) are the best choice models for semantic search

  • Type: Dense vector models.
  • Pros:
    • Rich semantic understanding.
    • Multilingual support.
    • Fine-tuning possible for domain-specific needs.
  • Popular Models:
    • msmarco-MiniLM-L-12-v3 – optimized for asymmetric search (short queries vs. long product descriptions) 3.
    • all-MiniLM-L6-v2 – fast and lightweight for general semantic tasks.
  • Use Case: Ideal for large-scale product catalogs and multilingual e-commerce platforms 3 4.

3. Hybrid Search (BM25 + Semantic)

  • Combine BM25 (keyword relevance) with semantic embeddings using Reciprocal Rank Fusion (RRF).
  • Delivers highly relevant results by balancing literal matches and contextual meaning 1.

The post Semantic Search appeared first on Customer Experience Management.

]]> Python for analysts most important datetime functions https://mietwood.com/python-for-analysts-most-important-datetime-functions https://mietwood.com/python-for-analysts-most-important-datetime-functions#comments Sun, 20 Jul 2025 16:18:38 +0000 https://mietwood.com/?p=3211 Python’s powerful date and time functions using the datetime and pandas libraries gives you a robust date table ready for Power BI and other business intelligence and analytical tools. Python for analysts most important datetime functions. Mastering Date and Time Functions in Python for Power BI Date Tables When working with Power BI, a well-structured...

The post Python for analysts most important datetime functions appeared first on Customer Experience Management.

]]>

Python’s powerful date and time functions using the datetime and pandas libraries gives you a robust date table ready for Power BI and other business intelligence and analytical tools. Python for analysts most important datetime functions.

Mastering Date and Time Functions in Python for Power BI Date Tables

When working with Power BI, a well-structured Date Table is essential for time intelligence calculations like YTD, QTD, MTD, and custom period comparisons. While Power BI has built-in date table features, using Python to generate a custom date table gives you full control over the structure, granularity, and logic.

In this post, we’ll explore Python’s powerful date and time functions using the datetime and pandas libraries, and show how to create a robust date table ready for Power BI.

Python for analyst – date and time functions – basics

Python provides the datetime module to work with dates and times. Here’s a quick overview:

from datetime import datetime, timedelta, date

# Current date and time
now = datetime.now()
print("Now:", now)

# Just the date
today = date.today()
print("Today:", today)

# Add 7 days
next_week = today + timedelta(days=7)
print("Next week:", next_week)

# Subtract 30 days
last_month = today - timedelta(days=30)
print("30 days ago:", last_month)

These functions are the foundation for generating date ranges and calculating custom columns like fiscal periods or holidays. Python for analysts most important datetime functions.

Creating a Date Range with Pandas

To build a date table, we need a continuous range of dates. pandas.date_range() is perfect for this:

import pandas as pd

# Generate a date range from 2020 to 2030
date_range = pd.date_range(start='2020-01-01', end='2030-12-31', freq='D')
df = pd.DataFrame({'Date': date_range})

This gives us a DataFrame with one row per day — the backbone of our date table.

Enriching the Date Table

Now let’s add useful columns for Power BI:

df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['MonthName'] = df['Date'].dt.strftime('%B')
df['Quarter'] = df['Date'].dt.quarter
df['Day'] = df['Date'].dt.day
df['Weekday'] = df['Date'].dt.weekday + 1  # Monday = 1
df['WeekdayName'] = df['Date'].dt.strftime('%A')
df['IsWeekend'] = df['Weekday'].isin([6, 7])
df['Week'] = df['Date'].dt.isocalendar().week
df['DayOfYear'] = df['Date'].dt.dayofyear

These columns allow for slicing and dicing your data in Power BI by year, month, weekday, and more.

Fiscal Calendar Support

Many businesses use fiscal calendars that don’t align with the calendar year. Here’s how to add a fiscal year starting in July:

df['FiscalYear'] = df['Date'].apply(lambda x: x.year if x.month < 7 else x.year + 1)
df['FiscalMonth'] = df['Date'].apply(lambda x: x.month - 6 if x.month >= 7 else x.month + 6)
df['FiscalQuarter'] = ((df['FiscalMonth'] - 1) // 3) + 1

This logic adjusts the fiscal year, month, and quarter based on a July start. Python for analysts most important datetime functions

5. Flags for Time Intelligence

Power BI benefits from flags that simplify DAX calculations:

today = pd.to_datetime('today').normalize()

df['IsToday'] = df['Date'] == today
df['IsCurrentMonth'] = (df['Date'].dt.month == today.month) & (df['Date'].dt.year == today.year)
df['IsCurrentYear'] = df['Date'].dt.year == today.year

You can also add flags for holidays, fiscal periods, or custom business logic. Python for analysts most important datetime functions

Exporting to CSV for Power BI

Once your date table is ready, export it:

df.to_csv('DateTable.csv', index=False)

You can now import this CSV into Power BI as a static date table.

7. Sample Output

Here’s a preview of what your date table might look like:

2025-01-0120251January1WednesdayFalse2025False
2025-07-0120257July3TuesdayFalse2026False
2025-12-25202512December4ThursdayFalse2026False

Pandas to_datetime() function

Python for analysts most important datetime functions – pandas

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  object        
 3   Dt_last_purchase        13483 non-null  object        

df_cust['Dt_first_purchase'] = pd.to_datetime(df_cust['Dt_first_purchase'],format="yyyy-mm-dd")
df_cust['Dt_last_purchase'] = pd.to_datetime(df_cust['Dt_last_purchase'],format="yyyy-mm-dd")

 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   CustomerId              13483 non-null  int64         
 1   Dt_first_rew_income     2987 non-null   datetime64[ns]
 2   Dt_first_purchase       13483 non-null  datetime64[ns]
 3   Dt_last_purchase        13483 non-null  datetime64[ns]

Advanced Tips

  • Holidays: Use external APIs or CSVs to mark public holidays.
  • Week Start: Adjust Weekday to match your locale (e.g., Monday vs. Sunday).
  • Time Zones: Use pytz or zoneinfo for timezone-aware datetime handling.
  • Dynamic Updates: Automate the script to regenerate the table monthly or yearly.

Certainly! Here’s a concise 400-word post on SQL Date and Time Functions, with examples, tailored for building a Date Table in Power BI:

SQL Date and Time Functions for Power BI Date Tables

When building reports in Power BI, a comprehensive Date Table is essential for enabling time-based calculations like YTD, MTD, and custom period comparisons. While Power BI can auto-generate a date table, using SQL to create one gives you full control over its structure and logic.

Let’s explore key SQL Server date and time functions and how to use them to build a robust date table.

1. Generating a Date Range

To create a date table, you need a continuous range of dates. In SQL Server, you can use a loop or a recursive CTE:

DECLARE @StartDate DATE = '2020-01-01';
DECLARE @EndDate DATE = '2030-12-31';

WITH DateCTE AS (
    SELECT @StartDate AS DateValue
    UNION ALL
    SELECT DATEADD(DAY, 1, DateValue)
    FROM DateCTE
    WHERE DateValue < @EndDate
)
SELECT * INTO DateTable FROM DateCTE
OPTION (MAXRECURSION 32767);

select * from DateTable

Here the example

Python for analysts most important datetime functions in sql
Python for analysts most important datetime functions in sql

This creates a table with one row per day between 2020 and 2030.

2. Adding Date Attributes

Once you have the base dates, enrich them with useful columns:

ALTER TABLE DateTable ADD 
    Year INT,
    Month INT,
    MonthName VARCHAR(20),
    Quarter INT,
    Weekday INT,
    WeekdayName VARCHAR(20);

UPDATE DateTable
SET 
    Year = YEAR(DateValue),
    Month = MONTH(DateValue),
    MonthName = DATENAME(MONTH, DateValue),
    Quarter = DATEPART(QUARTER, DateValue),
    Weekday = DATEPART(WEEKDAY, DateValue),
    WeekdayName = DATENAME(WEEKDAY, DateValue);

These columns allow for flexible filtering and grouping in Power BI.

3. Fiscal Calendar and Flags

You can also add fiscal logic and flags:

ALTER TABLE DateTable ADD FiscalYear INT;

UPDATE DateTable
SET FiscalYear = CASE 
    WHEN MONTH(DateValue) >= 7 THEN YEAR(DateValue) + 1
    ELSE YEAR(DateValue)
END;

Add flags like IsWeekend, IsToday, or IsCurrentMonth to simplify DAX expressions.

Conclusion

Python offers a flexible and powerful way to create a custom date table for Power BI. With just a few lines of code, you can generate a rich dataset that supports advanced time intelligence and reporting needs.

Whether you’re working with fiscal calendars, custom flags, or multilingual support, Python gives you the tools to tailor your date table exactly to your business requirements.

SQL’s date and time functions like DATEADD, DATEPART, DATENAME, and YEAR are powerful tools for building a custom date table. Once created, export it to Power BI or use it as a view for dynamic reporting.

Would you like a ready-to-run SQL script for a complete date table?

Find more resources in our course of Advanced programming for business analysts

The post Python for analysts most important datetime functions appeared first on Customer Experience Management.

]]> https://mietwood.com/python-for-analysts-most-important-datetime-functions/feed 2 Biłgoraj – Zalew Bojary – pływanie 100 km https://mietwood.com/bilgoraj-zalew-bojary-plywanie-100-km Sun, 13 Jul 2025 13:56:53 +0000 https://mietwood.com/?p=3173 Damian Błaszczyk będzie pływał w Zalewie Bojary, stawiając sobie za cel pokonanie 100 km. Biłgoraj – Zalew Bojary – pływanie 100 km. Przygotowaliśmy zadanie matematyczne jak może wyglądać takie pływanie i w jaki sposób pokonać 100km. Rozwiązanie zadania w dalszej części artykułu. Cel tego wydarzenia Ośrodek Sportu i Rekreacji w Biłgoraju “Na Fali” ma zaszczyt zaprosić na wyjątkowe wydarzenie,...

The post Biłgoraj – Zalew Bojary – pływanie 100 km appeared first on Customer Experience Management.

]]> Damian Błaszczyk będzie pływał w Zalewie Bojary, stawiając sobie za cel pokonanie 100 km. Biłgoraj – Zalew Bojary – pływanie 100 km. Przygotowaliśmy zadanie matematyczne jak może wyglądać takie pływanie i w jaki sposób pokonać 100km. Rozwiązanie zadania w dalszej części artykułu.

Cel tego wydarzenia

Ośrodek Sportu i Rekreacji w Biłgoraju “Na Fali” ma zaszczyt zaprosić na wyjątkowe wydarzenie, które po raz drugi zjednoczy nas wokół szczytnego celu! W dniach 25-27 lipca 2025 roku Zalew Bojary w Biłgoraju stanie się areną drugiej edycji akcji “Na Fali Nadziei – Przekraczając Granice”! – czytaj tutaj

W tym roku wspieramy Adasia Iwanejko, 12-latka z Woli Małej, który dzielnie walczy ze śmiertelną dystrofią mięśniową Duchenne’a. Jego jedyną szansą na powrót do zdrowia jest kosztowna terapia genowa w USA, której szacunkowy koszt przekracza 16 milionów złotych. Wierzymy, że dzięki wspólnej mobilizacji możemy zdziałać cuda i pomóc Adasiowi spełnić marzenie o normalnym życiu!

Zadanie 1. Biłgoraj – Zalew Bojary – pływanie 100 km

Pływak Damian Błaszczyk ma pokonać dystans 100 km pływając po okręgu przywiązany na linie o długości 100m do słupa o nieistotnej grubości. Lina obraca się wokół słupa zachowująć cały czas długość 100m. Zadanie to jest do wykonania na zalewie Bojary ponieważ zalew ten ma średnicę około 230m. Tak więc zakładamy, że pływak będzie okrążał zalew aż pokona dystans D=100km. Pytanie jest ile okrążeń musi wykonać pływak?

Obliczenia przedstawiono w tabeli poniżej. Pływak wykona 8 rund po 10 okrążeń w prawo, co daje 80 okrążeń i 80 okrążeń w lewo, co daje razem 160 okrążeń. Każde okrążenie daje dystans d=2Pir = 628 m. Tak więc jedna runda 10 okrążeń daje 6,280 m, 8 rund to 50,240 m. Przepłynięcie 100 km wymaga więc 16 rund czyli 160 okrążeń zalewu. Płynąc w tempie 4 km/h zajmuje czas około 25 godzin. W informacji prasowej jest wzmianka, że pływak zamierza pływać 48 godzin, więc jest to realny czas na pokonanie dystansu 100 km. Biłgoraj – Zalew Bojary – pływanie 100 km

Czas potrzebny na pokonanie dystansu 100 km obliczamy ze wzoru: t = d/v, gdzie d=dystans 100 km, a v=prędkość 4 km/godz. Przepłynięcie 100 km w tempie 4 km/h zajmuje czas około 25 godzin. Około oznacza to, że prędkość pływaka może być nieco większa lub mniejsza, wtedy ten czas ulegnie zmianie.

Zadanie 2

Teraz zakładamy, że średnicy słupa wynosi 20 cm, co jest bardziej realne. Lina nawija się na słup w ten sposób, że po każdych 10 ciu okrążeniach przeskakuje na kolejną warstwę zwiększając tym samym średnicę słupa. Pytanie: ile okrążeń musi wykonać pływak aby pokonać dystans 100 km? Biłgoraj – Zalew Bojary – pływanie 100 km.

Rozwiązanie zadania:

  • Długość początkowa liny = promień początkowy okręgu = 100 m.
  • Średnica liny = 1 cm ( rl = 0,01 m).
  • Średnica słupa = 20 cm (promień słupa = 10cm = r0 = 0,1m).

Etap 1, Pływak w pierwszym okrążeniu pokona dystans d = 2Pir = 628 m. W czasie tego okrążenia lina nawinie się na słup, co spowoduje, że jej długość zmniejszy się o d0 = 2Pi * r0 = 0,628 m. Tak więc drugie okrążenie będzie miało długość 628 m – 0,628 m = 627,2 m. Itd aż do 10-tego okrążenia. Czyli w okrążeniach 1-10, lina skraca się o d0 = 2*Pi*r0 = 6.28 * 0,1m = 0,628 m per okrążenie, czyli razem skróci się o 6,28m. Po 10 okrążeniach lina przeskakuje na wierzch poprzednio nawiniętej warstwy, co zwiększa promień słupa o grubość liny, czyli o 1 cm (0,01 m). Tak więc w okrążeniach 11-20, lina skraca się o d2 = 2 Pi * (r0+rl) = 6.28 * 0,11 m = 0,69 m.

Biłgoraj - Zalew Bojary - pływanie 100 km

Całość obliczeń przedstawia tabela poniżej. Pływak bedzie płynął na coraz to krótszej linie, co spowoduje, że ostatecznie cała lina zawinie się na słup. Po ilu okrążeniach to nastąpi i jaki dystans pływak pokona w tym czasie?

Jeśli pierwsze 10 okrążeń skraca line o 6,28, drugie 10 okrążeń skraca linę o 6,91 m itd. – patrz kolumnę czwartą Ubytek*10, to 100 m liny wystarczy na 10 rund (etapów) po 10 okrążeń i 11 rundę 6 okrążeń. Razem będzie to 106 okrążeń. Da to dystans 37,655 m. Jeśli pływak płynął w prawo to teraz powinien płynąć w lewo, co da kolejne 37,655 m. Ciągle nie będzie to pełne 100 km. Potrzebne są jeszcze 4 rundy o 8 okrążeń. Czyli podsumowując: pływak przepłynie łącznie 106 okrążeń w prawo i 106 w lewo a potem jeszcze 48 okrążeń w prawo. Razem da to 260 okrążeń. Z racji tego, że promień okręgu jest zmienny, co powoduje, że pokonywany dystans jest zmienny z każdym okrążeniem, to do pokonania 100 km potrzebne jest 260 okrążeń czyli o 100 wiecej niż w poprzedniej metodzie. Biłgoraj – Zalew Bojary – pływanie 100 km.

  • Dystans D okrążenia po okręgu o promieniu D = 2PiR, gdzie R = promień okręgu, w naszym przypadku jest to 100m, a 2*Pi = 6.28. Dystans okrążenia po okręgu 100m wynosi 628 m.
  • Jeśli lina nawija się na słup o promieniu r = 0,1 m, to podczas jednego okrążenia skraca się o 2Pir = 0,628 m. Biłgoraj – Zalew Bojary – pływanie 100 km.

Zadanie 3

Ciekawe zadanie powstaje, jeśli lina będzie się skracać za każdym razem nawijając się na poprzednie warstwy tworząc spiralę. W takiej spirali każde okrążenie jest krótsze o 0,628 m tworząc ciąg liczbowy. Do rozwiązanie tego zadania wykorzystaliśmy Google Gemini https://gemini.google.com , co przedstawia poniższy obrazek:

Biłgoraj - Zalew Bojary - pływanie 100 km

Równanie to można uprościć do następującej formy ( Biłgoraj – Zalew Bojary – pływanie 100 km )

i następnie

co po wymnożeniu daje równanie kwadratowe

i następnie

Równanie - Biłgoraj - Zalew Bojary - pływanie 100 km

W tej sytuacji mamy równanie kwadratowe typu ax^2 + bx + c =0. Używając formuły kwadratowej x = (−b ± pierwiastek kwadratowy z ( b2 − 4ac) ) / 2a​​, dostajemy: n = 160,43. Tak więc w takiej sytuacji pływak wykona 160 okrążenia na ciągle skracającej się linie. Biłgoraj – Zalew Bojary – pływanie 100 km

Dla miłośników Pythona, zadanie może być rozwiązane przy pomocy poniższej procedury. Biłgoraj – Zalew Bojary – pływanie 100 km.

def calculate_turns(rope_length, rope_thickness, bar_radius, shorten_per_turn, turns_per_layer, layer_increase):
    turns = 0
    while rope_length > 0:
        turns += 1
        rope_length -= shorten_per_turn
        if turns % turns_per_layer == 0:
            bar_radius += layer_increase / 2
            shorten_per_turn = 2 * 3.14 * (bar_radius + rope_thickness)
    return turns
# Given values
rope_length = 100  # meters
rope_thickness = 0.01  # meters (1 cm)
bar_radius = 0.1  # meters
shorten_per_turn = 0.628  # meters
turns_per_layer = 10
layer_increase = 0.02  # meters (2 cm)

# Calculate the number of turns
turns = calculate_turns(rope_length, rope_thickness, bar_radius, shorten_per_turn, turns_per_layer, layer_increase)

print(f"The swimmer can make {turns} turns before the rope ends.")

More python here

Biłgoraj – Zalew Bojary – pływanie 100 km

Pływania w dobrej sprawie – Biłgoraj – Zalew Bojary – pływanie 100 km

Artykuł opisuje niezwykłe wyzwanie wytrzymałościowe podjęte przez pływaka Damiana Błaszczyka, który postanowił przepłynąć 100 kilometrów w zbiorniku wodnym Zalew Bojary w Biłgoraju. Wydarzenie to, będące częścią inicjatywy „Na Fali Nadziei – Przekraczając Granice”, ma nie tylko sportowy, ale przede wszystkim charytatywny charakter. Celem tegorocznej edycji jest wsparcie Adasia Iwanejko, 12-letniego chłopca chorego na dystrofię mięśniową Duchenne’a. Zebrane środki mają pomóc w sfinansowaniu kosztownej terapii genowej w USA, której koszt przekracza 16 milionów złotych.

Matematyka pływania – modelowanie dystansu

Autor bloga przedstawia matematyczne modele opisujące, jak można przepłynąć 100 km w jednym zbiorniku. W pierwszym scenariuszu Damian pływa po idealnym okręgu, przymocowany do słupa liną o długości 100 metrów. Każde okrążenie ma wtedy około 628 metrów (obliczone ze wzoru na obwód koła: (2\pi r)). Aby osiągnąć 100 km, potrzeba 160 okrążeń, zmieniając kierunek co 10, by równomiernie obciążać ciało. Przy stałym tempie 4 km/h, całość zajęłaby około 25 godzin, choć wydarzenie przewiduje 48 godzin, co czyni cel realnym.

Realizm: lina nawijająca się na słup

Drugi scenariusz uwzględnia bardziej realistyczny aspekt – lina nawija się na słup, skracając z każdym okrążeniem promień pływania. Słup ma 20 cm średnicy, a lina 1 cm grubości. Po każdych 10 okrążeniach lina tworzy nową warstwę, zwiększając promień słupa. W efekcie każde kolejne okrążenie jest krótsze. Obliczenia pokazują, że Damian może wykonać 106 okrążeń w jednym kierunku, pokonując 37,7 km. Powtórzenie tego w przeciwnym kierunku daje kolejne 37,7 km. Aby osiągnąć 100 km, potrzeba jeszcze 48 dodatkowych okrążeń, co daje łącznie 260 okrążeń – o 100 więcej niż w pierwszym modelu.

Model spiralny – matematyczna elegancja

Trzeci model zakłada spiralne nawijanie liny, gdzie każde okrążenie jest krótsze o stałą wartość. Tworzy to ciąg arytmetyczny długości okrążeń. Korzystając z wzoru na sumę ciągu, autor oblicza, że potrzeba około 160,43 okrążeń, by osiągnąć 100 km. Model ten jest zbliżony do pierwszego, ale bardziej realistyczny i matematycznie elegancki.

Podsumowanie

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. Biłgoraj – Zalew Bojary – pływanie 100 km.

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. To także przykład, jak pasja i nauka mogą wspólnie służyć wyższemu celowi.

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. To także przykład, jak pasja i nauka mogą wspólnie służyć wyższemu celowi.

Artykuł łączy sport, charytatywność i matematykę w inspirującą opowieść. Pokazuje nie tylko niezwykłe wyzwanie fizyczne, ale też to, jak matematyka może pomóc zrozumieć i zaplanować realne działania. Wydarzenie nad Zalewem Bojary to dowód ludzkiej wytrwałości i solidarności – każdy przepłynięty metr to krok ku nadziei dla chorego chłopca. To także przykład, jak pasja i nauka mogą wspólnie służyć wyższemu celowi.

The post Biłgoraj – Zalew Bojary – pływanie 100 km appeared first on Customer Experience Management.

]]> Semantic Search with Elasticsearch https://mietwood.com/semantic-search-with-elasticsearch Fri, 11 Jul 2025 10:11:19 +0000 https://mietwood.com/?p=3158 Semantic search with Elasticsearch is must have for modern e-commerce. Elasticsearch is a powerful search engine, scalable data store, and vector database built on Apache Lucene. It’s optimized for speed and relevance on production-scale workloads. You can use Elasticsearch to index your product database and built beautiful Semantic search with Elasticsearch. How AI is changing...

The post Semantic Search with Elasticsearch appeared first on Customer Experience Management.

]]> Semantic search with Elasticsearch is must have for modern e-commerce. Elasticsearch is a powerful search engine, scalable data store, and vector database built on Apache Lucene. It’s optimized for speed and relevance on production-scale workloads. You can use Elasticsearch to index your product database and built beautiful Semantic search with Elasticsearch.

How AI is changing search behaviors

https://www.nngroup.com/articles/ai-changing-search-behaviors

Semantic search with Elasticsearch – intro

https://www.elastic.co/search-labs/blog/introduction-to-vector-search

Language model implementation

As the first we should initiate a language model and prepare product data for input. Semantic search with Elasticsearch require data indexing via vector transformer.

import pandas as pd
import numpy as np

# function to normalize vectors
def normalize_embedding(embedding):
    norm = np.linalg.norm(embedding)
    return embedding / norm if norm != 0 else embedding

# select products from sql database
def select_products():
    q = """
        SELECT 
         [ProductId]
        ,[ProdIdx]
        ,[ProductName]
        FROM [DB_Products] 
    """
    dfp = read_from_sql_server(q);
    return dfp

# import sentence transformers and initiate model
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
print(model)

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

model = model.to(device)
print(model)

Data indexing for semantic search

Now we can index product data into Elasticsearch database (index). We will index product names as vectors and lexically. This allows hybrid search. Semantic search with Elasticsearch.

# initiate Elasticsearch client
from elasticsearch import Elasticsearch, helpers
es = Elasticsearch('http://localhost:9100')

# check es client
from pprint import pprint
pprint(es.info().body)

# ReCreate the index with dense_vector and text mappings
# step 1
es.indices.delete(index="prod_search_hybrid", ignore_unavailable=True)

# step 2 - mappings
es.indices.create(
    index="prod_search_hybrid",
    mappings={
        "properties": {
            "embedding": {
                "type": "dense_vector",
                "dims": 384,
                "index": True,
                "similarity": "cosine"
            },
            "prodct_name": {
                "type": "text"
            }
        }
    },
)

# list indices and number of documents indexed 
def indices_list():
    indices = es.cat.indices(format='json')
    return [x['index'] for x in indices]
# -------------------
print(indices_list())
# ------------------------------

# Create documents for embedding
documents = []
for i, r in df_docs.iterrows():
    documents.append({        
        'product_name': r['ProductName'][:256].lower(),        
    })

print(f' Created table of {len(documents)} docs')

# Prepare bulk operations
from tqdm import tqdm                                         # for a prograss bar
operations = []
for document in tqdm(documents, total=len(documents)):
    operations.append({'index': {'_index': 'prod_search_hybrid'}})
    operations.append({
        **document,
        'embedding': get_embedding(document['product_name']), # vectors for semantic search
        'product_name': document['product_name']              # the text field for hybrid search
    })

# Bulk insert the data into Elasticsearch
response = es.bulk(operations=operations)
print(f' Records indexed: {len(response["items"])}')

Hybrid search

For hybrid search we combine match and knn search inside a bool query. The _name field return what what part of the query has returned results. This allow to build hybride scoring. As you can see we vectorize query_h to query_vector using the same procedure get_embeding as we were using during indexing. Semantic search with Elasticsearch.

query_h = "anti-explosion device"

# Print the query vector for debugging
# query_vector = get_embedding(query_h)
# print("Query Vector:", query_vector)

response = es.search(
    index='prod_search_hybrid',
    body={
        "query": {
            "bool": {
                "should": [
                    {
                        "match": {
                            "product_name": {
                                "query": query_h,
                                "_name": "text_match"
                            }
                        }
                    },
                    {
                        "knn": {
                            "field": "embedding",
                            "query_vector":  query_vector,
                            "k": 10,
                            "num_candidates": 100,
                            "_name": "semantic_search"
                        }
                    }
                ]
            }
        }
        ,
        'size': 30
    }
)

After that we can extract product names and scoring and build a list according to our intention. In here we separate products according to field _name and then build list of top 10 lexical match and semantic similarity. More about similarity measures you can read in post Measuring product similarity

# Extract product names, scores, and sources
products = [
    (hit["_source"]["product_name"], hit["_score"], hit.get("matched_queries", []))
    for hit in response["hits"]["hits"]
]

# Separate products into text_match and semantic_search groups
text_match_products = [product for product in products if "text_match" in product[2]]
semantic_search_products = [product for product in products if "semantic_search" in product[2]]

# Sort each group by score in descending order
sorted_text_match_products = sorted(text_match_products, key=lambda x: x[1], reverse=True)[:10]
sorted_semantic_search_products = sorted(semantic_search_products, key=lambda x: x[1], reverse=True)[:10]

# Print top 10 text_match products
print("\nTop 10 Text Match Products:")
for product in sorted_text_match_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

# Print top 10 semantic_search products
print("\nTop 10 Semantic Search Products:")
for product in sorted_semantic_search_products:
    print(f"Product Name: {product[0]}, Score: {product[1]}")

Full-text search, also known as lexical search, is a technique for fast, efficient searching through text fields in documents. Documents and search queries are transformed to enable returning relevant results instead of simply exact term matches. Fields of type text are analyzed and indexed for full-text search.

You can combine full-text search with semantic search using vectors to build modern hybrid search applications. While vector search may require additional GPU resources, the full-text component remains cost-effective by leveraging existing CPU infrastructure.

Another example of vector indexing and sementic search you can find here: https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb

Vector search setup and performing hybrid search

https://www.elastic.co/search-labs/blog/vector-search-set-up-elasticsearch

https://www.elastic.co/search-labs/blog/hybrid-search-elasticsearch

Filtering. Semantic search with Elasticsearch

Filter context is mostly used for filtering structured data. For example, use filter context to answer questions like:

  • Does this timestamp fall into the range 2015 to 2016?
  • Is the status field set to “published”?

Filter context is in effect whenever a query clause is passed to a filter parameter, such as the filter or must_not parameters in a bool query. Learn more about filter context in the Elasticsearch docs.

Keyword Filtering

This is an example of adding a keyword filter to the query. The example retrieves the top books that are similar to “javascript books” based on their title vectors, and also Addison-Wesley as publisher. Semantic search with Elasticsearch.

response = client.search(
    index="book_index",
    knn={
        "field": "title_vector",
        "query_vector": model.encode("javascript books"),
        "k": 10,
        "num_candidates": 100,
        "filter": {"term": {"publisher.keyword": "addison-wesley"}},
    },
)

pprint(response)

Query and filter context

Relevance scores

By default, Elasticsearch sorts matching search results by relevance score, which measures how well each document matches a query. The relevance score is a positive floating point number, returned in the _score metadata field of the search API. The higher the _score, the more relevant the document. While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a query or filter context.

Query context

In the query context, a query clause answers the question How well does this document match this query clause? Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the _score metadata field. Query context is in effect whenever a query clause is passed to a query parameter, such as the query parameter in the search API. Semantic search with Elasticsearch

Filter context

A filter answers the binary question “Does this document match this query clause?”. The answer is simply “yes” or “no”. Filtering has several benefits:

  1. Simple binary logic: In a filter context, a query clause determines document matches based on a yes/no criterion, without score calculation.
  2. Performance: Because they don’t compute relevance scores, filters execute faster than queries.
  3. Caching: Elasticsearch automatically caches frequently used filters, speeding up subsequent search performance.
  4. Resource efficiency: Filters consume less CPU resources compared to full-text queries.
  5. Query combination: Filters can be combined with scored queries to refine result sets efficiently.

Filters are particularly effective for querying structured data and implementing “must have” criteria in complex searches.

Structured data refers to information that is highly organized and formatted in a predefined manner. In the context of Elasticsearch, this typically includes:

  • Numeric fields (integers, floating-point numbers)
  • Dates and timestamps
  • Boolean values
  • Keyword fields (exact match strings)
  • Geo-points and geo-shapes

Unlike full-text fields, structured data has a consistent, predictable format, making it ideal for precise filtering operations. Semantic search with Elasticsearch.

Common filter applications include:

  • Date range checks: for example is the timestamp field between 2015 and 2016
  • Specific field value checks: for example is the status field equal to “published” or is the author field equal to “John Doe”

Filter context applies when a query clause is passed to a filter parameter, such as:

Filters optimize query performance and efficiency, especially for structured data queries and when combined with full-text searches.

GET /_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title":   "Search"        }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [
        { "term":  { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}

Read more: Product Search and Product classification for E-commerce

Reciprocal rank fusion

Reciprocal rank fusion (RRF) is a method for combining multiple result sets with different relevance indicators into a single result set. RRF requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results. Semantic search with Elasticsearch.

RRF uses the following formula to determine the score for ranking each document:

score = 0.0
for q in queries:
    if d in result(q):
        score += 1.0 / ( k + rank( result(q), d ) )
return score

# where
# k is a ranking constant
# q is a query in the set of queries
# d is a document in the result set of q
# result(q) is the result set of q
# rank( result(q), d ) is d's rank within the result(q) starting from 1

You can use RRF as part of a search to combine and rank documents using separate sets of top documents (result sets) from a combination of child retrievers using an RRF retriever. A minimum of two child retrievers is required for ranking.

Semantic search with Elasticsearch. RRF retriever combined the results.
Semantic search with Elasticsearch. RRF retriever combined the results.

We rank the documents based on the RRF formula with a rank_window_size of 5 truncating the bottom 2 docs in our RRF result set with a size of 3. We end with _id: 3 as _rank: 1, _id: 2 as _rank: 2, and _id: 4 as _rank: 3. This ranking matches the result set from the original RRF search as expected.

In this example, we execute the knn and standard retrievers independently of each other. Then we use the rrf retriever to combine the results.

  1. First, we execute the kNN search specified by the knn retriever to get its global top 50 results.
  2. Second, we execute the query specified by the standard retriever to get its global top 50 results.
  3. Then, on a coordinating node, we combine the kNN search top documents with the query top documents and rank them based on the RRF formula using parameters from the rrf retriever to get the combined top documents using the default size of 10.

Note that if k from a knn search is larger than rank_window_size, the results are truncated to rank_window_size. If k is smaller than rank_window_size, the results are k size.

GET example-index/_search
{
    "retriever": {
        "rrf": {
            "retrievers": [
                {
                    "standard": {
                        "query": {
                            "term": {
                                "text": "shoes"
                            }
                        }
                    }
                },
                {
                    "knn": {
                        "field": "vector",
                        "query_vector": [1.25, 2, 3.5],
                        "k": 50,
                        "num_candidates": 100
                    }
                }
            ],
            "rank_window_size": 50,
            "rank_constant": 20
        }
    }
}

The post Semantic Search with Elasticsearch appeared first on Customer Experience Management.

]]> Survival Analysis Models https://mietwood.com/survival-analysis-models Wed, 02 Jul 2025 10:30:53 +0000 https://mietwood.com/?p=3115 Survival analysis is a statistical method used to modeling object behavior dependent on set of variables (x1 .. xn) in time-to-event period. It is especially useful in modeling the probability of object’s survival in certain circumstances. One can analyze a timeline to events occurrences in relation to variables influencing the time until a specific event occurs like...

The post Survival Analysis Models appeared first on Customer Experience Management.

]]> Survival analysis is a statistical method used to modeling object behavior dependent on set of variables (x1 .. xn) in time-to-event period. It is especially useful in modeling the probability of object’s survival in certain circumstances. One can analyze a timeline to events occurrences in relation to variables influencing the time until a specific event occurs like death, failure, or customer say “Goodby to the company”. Here’s a breakdown of key terminology of survival analysis.

Key terminology of survival analysis models

  • Event: The outcome of interest, such as death, disease occurrence, customer churn, or equipment failure.
  • Time: The duration from a defined starting point (e.g., start of treatment or customer acquisition) to the occurrence of the event or the end of observation (censoring).
  • Censoring: When the event of interest is not observed for some individuals during the study period, making their exact survival time unknown.
  • Survival Function S(t): The survival function S(t) is defined as the probability that a subject survives (i.e., does not experience the event) beyond time t.
  • Hazard Function h(t): The hazard function h(t) represents the instantaneous rate at which the event occurs at time t, given that the subject has survived up to that time.
  • Linking the Survival and Hazard Functions: The survival and hazard functions are mathematically related. Knowing one allows you to derive the other, reflecting the duality between retention (survival) and churn (event occurrence).

Linking the Survival and Hazard Functions

The survival and hazard functions are mathematically related. Knowing one allows you to derive the other, reflecting the duality between retention (survival) and churn (event occurrence).

In summary, the cumulative hazard rate of subject i at time t can also be defined as the negative logarithm of the survival function at time t.

Stratification

When we have a variable influence in some moment of time, we can divide the period of observation to two periods, when the time before the variable application and the the time post. It constitutes a new starting point of the new treatment application to the patient. The shift of the survival function could be observed.

Kaplan-Meier Estimator

Kaplan-Meier Estimator is a non-parametric model for estimating the survival function, often used as a first step in survival analysis. It demonstrate the probability of survival a certain time in function of time. The maximum value is 1 and the K-M curve goes approximately to zero when time increases.

Cox Proportional Hazards Model

Cox Proportional Hazards Model is a semi-parametric model that incorporate the influence of some predictor variables to the hazard rate or survival model.  Each subject has an observed survival time 0-t and an event variable xE that shows whether the event has occurred. It is also observed in time 0-t tat variables x1 .. n have had an influence on event xE occurrence. The Cox model incorporate the influence of x1 .. xn factors to estimation the time of survival or hazard rate as n explanatory variables. This make the model can also predict a time of survival.

Disclaimer: the post is inspired by: Benjamin Lee, A Comparison Study of Parametric and Machine Learning Survival Analysis Models to Predict Customer Churn in the Edtech Sector, Vienna, 27th January, 2025
Benjamin Lee, Peter Filzmoser, link: here

Dataset for Cox Proportional Hazards Model

If we have dataset like here

Survival Analysis Models. The general data input structure.

The Kaplan-Meier estimator is the statistical tool used to estimate a true survival function from available data and can be considered the ’best’ estimator of survival probability when no parametric structure is assumed. This estimator is a non-parametric estimator that only requires the time-to-event (or time-to-censoring) t , and the event status e for every subject. With this information, the survival function estimator S(t) is given by:

survival function estimator

where ej is the time at which at least one event e occurred, and nj is the total number of subjects who have been censored or have not had the event yet at time tj .

One of the main drawbacks of the Kaplan-Meier model is that it is not able to take any subject covariates into account – what means, it does not provide explanatory power in terms of different variables within a group.

Understanding the Cox Proportional Hazards Model

When we want to understand how factors like age, treatment type, or blood pressure affect survival time, the Cox Proportional Hazards model is a go-to tool. Instead of trying to pin down the exact risk at every single moment, this model cleverly separates the hazard rate into two distinct parts.

The core idea is to model how specific factors (or covariates) have a multiplicative effect on an underlying hazard rate. The formula for the model looks like this:

h(t,X)=h0​(t)⋅exp(βX)

Let’s break that down:

  • h0​(t) is the Baseline Hazard. Think of this as the underlying risk of the event for a “standard” individual (where all covariate values are zero) over time. It’s the part of the model that changes with time (t), but it’s “non-parametric,” meaning we don’t assume its shape. We let the data speak for itself.
  • exp(βX) is the Covariate Effect. This is the parametric part of the model and it’s where your specific factors come in.
    • X is the set of your covariates (e.g., age, treatment type, or blood pressure).
    • β are the coefficients, similar to those in a linear regression, that the model estimates.
    • This component tells us how much an individual’s unique characteristics increase or decrease their risk compared to the baseline. Applying the exponential function, exp(), ensures this multiplier is always positive, as negative risk doesn’t make sense.

Notice that only the baseline hazard depends on time. The covariate effect, exp(βX), is constant over time. This separation is the key to the model’s power and leads directly to its most important assumption.

In the Cox proportional hazards model, a covariate is a predictor variable that you include in the model to explain or predict the risk (hazard) of a specific event occurring over time, such as death, failure, or relapse. These covariates can be continuous (e.g., age, blood pressure) or categorical (e.g., treatment group, gender).

What the program calculates:

  • The model estimates the effect of each covariate on the hazard function—the instantaneous risk of the event happening at a certain time.
  • It computes coefficients (often denoted as ββ) for each covariate, quantifying how the risk changes with a one-unit increase in that variable, assuming other covariates remain constant.
  • These coefficients translate into hazard ratios (exponentiated coefficients), which tell how much the hazard (risk) is multiplied by when the covariate changes by one unit.
  • The baseline hazard function h0(t)h0(t) represents the hazard if all covariates were zero; the model then scales this baseline hazard depending on the values of the covariates for each individual.
  • Overall, the Cox model evaluates how your covariates modify the risk of the event over survival time, assuming that these effects multiply the baseline hazard proportionally and do not change with time (proportional hazards assumption).

Formally, the hazard function is modeled as

where x1,x2,…,xpx1,x2,…,xp are your covariates and β1,β2,…,βpβ1,β2,…,βp are their estimated effects.

Summary:

  • Your covariate is any variable in your data that may affect the timing of the event.
  • The Cox model calculates how these covariates affect the hazard or instantaneous risk of the event happening, summarized through hazard ratios.
  • This helps you understand which factors increase or decrease risk while accounting for the survival time and censoring in your dataset.

This explanation aligns with standard survival analysis literature and is consistent with how lifelines or R coxph functions use covariates in the Cox model.

The “Proportional” in Proportional Hazards

The model’s name comes from its core assumption: the effect of the covariates is constant over time. In other words, the hazard ratio between any two individuals remains proportional throughout the entire timeline.

Let’s make this concrete. Imagine we have a single covariate, x – age, and two subjects, ‘a’ and ‘b’. Their hazard rates are:

  • Subject a: h(t,xa​)=h0​(t)⋅exp(βxa​)
  • Subject b: h(t,xb​)=h0​(t)⋅exp(βxb​)

If we look at the ratio of their hazards, the baseline hazard h0​(t) cancels out:

As you can see, time (t) has vanished from the right side of the equation. This means that if subject ‘a’ has double the risk of subject ‘b’ on day 1, they will also have double the risk on day 100 or day 500. The risk difference is because of age difference. Their relative risk is constant, or proportional. This powerful assumption is a direct consequence of the model’s structure.

If one has a dataset which violates the proportional hazards assumption, it causes reduction in predictive power of the model. The hazard as an output from the model is a useful tool in assessing the general magnitude of a covariates’ effect on the time of survival. One can interpret the hazard ratio as the weighted average of true hazard ratios over the time period. Therefore, one must not strictly conform to the proportional hazards assumption, but always check if dataset is appropriate to this assumption.

You can check if your dataset meets the proportional hazards assumption of the Cox model using both visual plots and formal statistical tests.

Visual Inspection (Graphical Methods)

Visual checks are often the first and most intuitive step.

Log-Log Survival Plots

This is a classic method for categorical covariates (like “treatment group” vs. “control group”).

  • How it works: You plot the logarithm of time, log(t), on the x-axis against a special transformation of the survival probability, -log(-log(S(t))), on the y-axis for each group.
  • What to look for: If the proportional hazards assumption holds, the resulting curves for each group should be roughly parallel and not cross. If the lines cross or move closer or further apart in a systematic way, the assumption may be violated.

Schoenfeld Residuals Plots

This is the most common and powerful visual method, and it works for both continuous and categorical covariates.

  • How it works: For each event that occurs, a residual is calculated that represents the difference between the observed covariate value and the expected covariate value for the individual who had the event. You then plot these residuals against time.
  • What to look for: If the assumption holds, you should see a random scatter of points around a horizontal line at zero. If you see any clear pattern or trend (e.g., a line with a positive or negative slope), it suggests the effect of the covariate changes over time, violating the assumption.

Formal Statistical Tests

A formal test gives you a p-value to help you decide if any violations you see are statistically significant.

  • How it works: The most common test is based on the Schoenfeld residuals. It formally tests whether the slope of a line fitted to the Schoenfeld residuals-vs-time plot is significantly different from zero. This is often done using a function like cox.zph() in R or the check_assumptions() method in Python’s lifelines library.
  • How to interpret the results:
    • Null Hypothesis (H_0): The effect of the covariate is constant over time (the proportional hazards assumption holds).
    • A low p-value (e.g., < 0.05) suggests you should reject the null hypothesis. This is evidence that the assumption is violated for that specific covariate.
    • A high p-value means you fail to reject the null hypothesis, so it’s reasonable to assume the proportional hazards assumption is met.

Example of check_assumptions(data) in Python

The dataset includes:

  • Subject ID
  • Survival time in days
  • Event status (1 = event occurred, 0 = censored)
  • Age group (1 – 4)

To use check_assumptions() from lifelines, you need to fit a Cox Proportional Hazards model, which requires at least one covariate (a variable that might affect survival, ex age group).

from lifelines import CoxPHFitter

# Fit the Cox model
cph = CoxPHFitter()
cph.fit(data, duration_col="Survival_in_days", event_col="Status_bool")

# Check proportional hazards assumption
cph.check_assumptions(data)
# or
cph.check_assumptions(data, p_value_threshold=0.05)

What to Do If the Assumption is Violated

If you find that a key variable violates the assumption, you have options:

  1. Stratification: You can stratify your model by the problematic covariate. This allows the baseline hazard function to be different for each level of that variable, resolving the violation. For example, if “gender” violates the assumption, you can stratify by it.
  2. Use Time-Dependent Covariates: You can modify the model to include an interaction term between the covariate and time. This explicitly models how the covariate’s effect changes over time.
  3. Choose a Different Model: If the violation is severe, a parametric model (like a Weibull or Log-Logistic model) that has a built-in shape for the hazard function might be a more appropriate choice for your data.

Parametric Models

When the underlying probability distribution of the dataset is known, one can use parametric models to model the survival function of the dataset. Once the underlying model is specified either in terms of the survival times or the logarithm of survival times, the model can be fitted and estimated using the maximum likelihood estimator.

Of course. Parametric models in survival analysis assume that a subject’s survival time follows a specific, known statistical distribution (like the exponential or Weibull distribution). Unlike non-parametric models (e.g., Kaplan-Meier) which don’t make assumptions about the data’s distribution, parametric models are defined by a set of parameters that dictate the shape of the survival and hazard functions.

Understanding them means understanding the hazard function each model assumes. The hazard function describes the instantaneous risk of an event occurring at a specific time, given that it hasn’t occurred yet.

Exponential Model

This is the simplest parametric model. It assumes the hazard rate is constant over time.

  • Core Idea: The risk of the event happening is the same every single day. If a component hasn’t failed by day 10, its risk of failing on day 11 is the same as it was on day 2.
  • Hazard Function: h(t)=λ (a constant)
  • How to Understand It: This model is defined by a single parameter, λ (lambda), the hazard rate. It’s best suited for events that don’t have a “memory” or aging process, such as the failure rate of certain electronic components or the occurrence of random external events.

Weibull Model

The Weibull model is a more flexible and widely used model because it does not assume a constant hazard rate.

  • Core Idea: The risk of the event can increase, decrease, or remain constant over time. This makes it much more adaptable to real-world scenarios.
  • Hazard Function: h(t)=λk(λt)k−1
  • How to Understand It: The model is defined by two main parameters:
    • λ (scale parameter): Stretches or compresses the curve.
    • k (shape parameter): This is the key. It dictates the nature of the hazard.
      • If k>1, the hazard increases over time (e.g., aging, where the risk of failure grows).
      • If k<1, the hazard decreases over time (e.g., post-surgery recovery, where the initial risk is high but drops).
      • If k=1, the Weibull model simplifies to the Exponential model with a constant hazard.

Log-Normal and Log-Logistic Models

These models are useful for situations where the hazard rate is not monotonic (i.e., it doesn’t just go up or down).

  • Log-Normal Model
    • Core Idea: Assumes that the logarithm of the survival time follows a normal (bell-shaped) distribution.
    • Hazard Function: The hazard rate first increases to a peak and then decreases.
    • How to Understand It: Think of situations where failure is most likely after a certain “wear-in” period, but if the subject survives past that peak, the immediate risk then declines. It’s often used in engineering for component fatigue.
  • Log-Logistic Model
    • Core Idea: Similar to the Log-Normal model, it also assumes the logarithm of survival time follows a specific distribution (the logistic distribution).
    • Hazard Function: The hazard rate can be hump-shaped (increasing then decreasing) or monotonically decreasing, depending on its parameters. It’s more flexible than the Log-Normal model.
    • How to Understand It: This model is popular in medical research, especially when studying diseases like cancer where the risk of mortality might peak some time after diagnosis and then fall. It’s also notable because its survival function has a simple, explicit formula, which makes interpreting odds easier.

Accelerated Failure Time (AFT) models

In contrast to the Cox model, AFT models assume that covariates are proportional with respect to survival time.

Metrics for Survival Analysis

Log-Rank Test

The log-rank test is used to compare two or more survival functions with each other. In this sense, it is analogous to the t-test or Pearson’s chi-squared test for survival analysis. Like those tests, the log-rank test tests the null hypothesis H0 that there is no difference between the survival functions being compared in the probability of an event e occurring at any time t.

The Log-Rank Test is a statistical test used in survival analysis to compare the survival distributions of two or more groups. It’s particularly useful for determining if there are significant differences in the time it takes for an event (like death, failure, or relapse) to occur between groups.

Example Calculation

Let’s consider a clinical trial comparing the survival times of patients using two different cancer treatments, Drug A and Drug B. Here’s a simplified dataset:

Time (months)Event (1=death, 0=censored)Group (A/B)
21A
30A
41B
61A
70B
81B

Steps to Calculate the Log-Rank Test:

  1. Combine the Data: List all unique time points from both groups.
  2. Calculate the Number at Risk: For each time point, determine how many patients are still at risk in each group.
  3. Calculate the Expected Events: For each time point, calculate the expected number of events (deaths) for each group.
  4. Compute the Test Statistic: Use the observed and expected events to calculate the test statistic.

Detailed Calculation:

  1. Combine the Data:
    • Time points: 2, 3, 4, 6, 7, 8
  2. Number at Risk:
    • At time 2: Group A: 2, Group B: 3
    • At time 3: Group A: 1, Group B: 3
    • At time 4: Group A: 1, Group B: 2
    • At time 6: Group A: 1, Group B: 2
    • At time 7: Group A: 0, Group B: 2
    • At time 8: Group A: 0, Group B: 1
  3. Expected Events:
    • At time 2: Expected events for Group A = (2/5) * 1 = 0.4, Group B = (3/5) * 1 = 0.6
    • At time 4: Expected events for Group A = (1/3) * 1 = 0.33, Group B = (2/3) * 1 = 0.67
    • At time 6: Expected events for Group A = (1/3) * 1 = 0.33, Group B = (2/3) * 1 = 0.67
    • At time 8: Expected events for Group A = 0, Group B = 1
  4. Test Statistic:
    • Sum the observed and expected events for each group.
    • Calculate the test statistic using the formula:
    $ \chi^2 = \sum \frac{(Oi – Ei)^2}{E_i} $ Where ( Oi ) is the observed number of events and ( Ei ) is the expected number of events. For our example, the test statistic would be calculated based on the observed and expected events at each time point.

If the test statistic is greater than the critical value from the chi-square distribution table (with 1 degree of freedom), we reject the null hypothesis and conclude that there is a significant difference between the survival distributions of the two groups [1] [2].

Example of Survival Analysis Models in Python

The code demonstrate a plot of survival and hazard functions.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# --- 1. Create a Sample Dataset ---
data_dict = {
    'time': [6, 7, 10, 15, 18, 22, 25, 30, 32, 38, 40, 45],
    'event': [1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1]  # 1 = event, 0 = censored
}
data = pd.DataFrame(data_dict)
data = data.sort_values(by='time').reset_index(drop=True)

# --- 2. Calculate Kaplan-Meier Survival Function ---
km_data = data.copy()
unique_times = sorted(km_data['time'].unique())
survival_prob = 1.0
results = []

for t in unique_times:
    at_risk = (km_data['time'] >= t).sum()
    events = km_data[(km_data['time'] == t) & (km_data['event'] == 1)].shape[0]
    
    if at_risk > 0:
        survival_prob *= (1 - events / at_risk)
    
    results.append({'time': t, 'survival': survival_prob})

km_results = pd.DataFrame(results)

# --- 3. Plot the Survival Function ---
plt.figure(figsize=(10, 6))
plt.step(km_results['time'], km_results['survival'], where='post', label='Kaplan-Meier Estimate')
plt.scatter(data[data['event'] == 0]['time'], km_results.loc[data[data['event']==0].index, 'survival'],
            marker='+', color='red', s=100, label='Censored')
plt.title('Survival Function (Kaplan-Meier Estimate)')
plt.xlabel('Time (Months)')
plt.ylabel('Survival Probability')
plt.ylim(0, 1.05)
plt.xlim(0, max(data['time']) + 5)
plt.grid(True)
plt.legend()
plt.savefig('manual_survival_function_plot.png')
plt.close()

# --- 4. Calculate Nelson-Aalen Cumulative Hazard ---
na_data = data.copy()
hazard = 0.0
na_results_list = []

for t in unique_times:
    at_risk = (na_data['time'] >= t).sum()
    events = na_data[(na_data['time'] == t) & (na_data['event'] == 1)].shape[0]

    if at_risk > 0:
        hazard += events / at_risk
    
    na_results_list.append({'time': t, 'cumulative_hazard': hazard})

na_results = pd.DataFrame(na_results_list)

# --- 5. Plot the Cumulative Hazard Function ---
plt.figure(figsize=(10, 6))
plt.step(na_results['time'], na_results['cumulative_hazard'], where='post', label='Nelson-Aalen Estimate')
plt.title('Cumulative Hazard Function (Nelson-Aalen Estimate)')
plt.xlabel('Time (Months)')
plt.ylabel('Cumulative Hazard')
plt.xlim(0, max(data['time']) + 5)
plt.grid(True)
plt.legend()
plt.savefig('manual_cumulative_hazard_plot.png')
plt.close()

[1]: DATAtab [2]: Real Statistics Using Excel

References

[1] Log-Rank Test: A Beginner’s Guide – DATAtab

[2] Log-Rank Test – Real Statistics Using Excel

The post Survival Analysis Models appeared first on Customer Experience Management.

]]>