DP2 Beginner’s Guide

Welcome to DP2 Beginner's Guide!

DP2 is a web data collection tool that helps you efficiently extract required data from websites.

This guide will teach you the complete process of configuring and using DP2 for web data collection.

1. Introduction

• 1. Introduction to DP2

2. System Architecture

• DP2 System Architecture and Component
  • Study
  • Jexter
  • Step
  • Project
  • Task
  • Elements
  • Worker Node
  • Data Input (Data In) & Data Output (Data Out)
  • XPath
  • JMESPath
  • API (Application Programming Interface)
  • Special Table Data Processing
  • Attachment Handling and Storage
  • Data Updates and Notifications

3. User Interface

• DP2 User Interface Overview

4. Steps and Extraction Guides

• Step Configuration ① - Category Step
  • Configuring category_step
• Jexter Configuration - Extract Category Information
  • Extracting the Category in category_step
• Step Configuration ② - Total Page Step
  • Configuring totalpage_step
• Jexter Configuration - Get Total Pages
  • Using Jexter to Extract the Total Number of Pages in totalpage_step
• Step Configuration ③ - List Step
  • Configuring List Steps (list_step)
  • Common Error Cases
• Jexter Configuration - Extract List Page Information
  • Extract Page Information in the list_step
• Step Configuration ④ - Detail Step
  • Configuring Detail Steps (detail_step)
• Extracting Drug Information in detail_step
  • Description
  • Detailed Case Explanation
  • Jexter Configuration：
  • Explanation of Configuration
• Step Configuration ⑤ - Attachment Step
  • Configuring the Attachment Handling Step (attachment_step)

5. API Configuration

• Configuring API Settings to Save Data
  • Define the API Endpoint
  • Specify the Table Name
  • Set a Unique Identifier
  • Configure Data Fields
  • Choose the Data Operation Type
  • Handle Attachments
  • Sample Configuration: MySQL
  • Sample Configuration: MongoDB
  • Handling Attachments in MongoDB
  • Summary

6. Post Configuration

• Example Overview
• Configuring Basic Information
• Setting Request Method and Data
• Defining Cookies and Headers
• Additional Configurations
• Complete Configuration Example

7. Data Management

• Batch Processing, Deletion, and Export
• Understanding Data Flow and Task Management
• Fundamentals of MongoDB Querying for DP2

8. Monitoring and Logging

• Monitoring and Logging DP2 Activities

9. Real-world Applications

• Real-world Scenarios and Use Cases

10. Simplifying Data Extraction with Jexter

• Simplifying Data Extraction with Jexter I
• Simplifying Data Extraction with Jexter：Parent
  • Common Forms of XPath Expressions Include:
  • Simplifying XPath Using Parent Configuration
  • Example 1: Nested Structure of Medicine Information
  • Example 2: Columnar Display of Medicine Components and Effects
  • Example 3: Medicine Reviews and User Ratings Combination
  • Example 4: Medicine Instructions and Warning Information
  • Example 5: Multi-Functional Medicine Page
  • Conclusion
• Simplifying Data Extraction with Jexter III
• Total Row: Defining the Total Number of Drugs to Extract
• Parent: Streamlining the XPath Configuration
• Elements: Specifying Drug Information to Extract
• Combining the Three Aspects for Efficient Drug Data Extraction
• prefix, postfix, and default
  • HTML Example
  • prefix and postfix
  • Application Example
  • Jexter Configuration
  • Expected Result
  • default
  • Application Example
  • Jexter Configuration Adjustment
  • Expected Result
• Execution Order of Extraction with the Jexter in DP2
  • Data Location
  • Specific Extraction Rules
  • Execution Order of Extraction Logic
  • Data Processing Functions
  • Other Key Concepts
• Comprehensive Example
• Conclusion

11. Querying Techniques

• Locating Data in DP2 Using XPath
  • Basics of XPath
  • Advanced Usage
  • Applying XPath in DP2
  • Extracting Drug Information in detail_step：
  • Extract the Total Number of Pages in totalpage_step：
  • Extracting the Category in category_step：
  • Extract Page Information in list_step：
  • Resource Link
  • Summary
• jq and JMESPath in DP2
  • Application of jq in DP2
  • Application of JMESPath in DP2
  • Combining jq and JMESPath
  • Learning Resources

12. Tips and Troubleshooting

• Tricks and Tips for DP2 (Continuously Updated)
• Troubleshooting and Frequently Asked Questions (FAQ) in DP2

13. Special Case Collection

• Special Cases: Avoiding Duplicate Keys
  • Common Issue: Using Duplicate Key Names
  • Using Nested Structures and the default Key
  • Conclusion
• Special Case： Extracting ‘category_id’ from DP2 Jexter System
• Special Case: Iterative Parameter Configuration
• Select Attachments
  • Jexter Configuration and Extraction Results
  • Conclusion
• Select Drug Category Links
  • Sample Web Page Structure
  • XPath Expression
  • Data Extraction Configuration
  • Example Extraction Results
  • Result Explanation