Bulk Disbursement Integration

Bulk Payment Design
Provide scalable and fault tolerant design with easy integration
Allow easy plugging with different payment types (Mojaloop, GSMA, Afrimoney and others)
Enable reuse of existing BPMNs to facilitate transfers
Prioritizing New Use Cases
G2P
P2G
Bulk Processor Flow
Bulk processing is designed as a separate system within Payment Hub EE architecture supporting
File integrity check with MD5/SHA256 checksum when received in channel connector
Bulk processor
Split streamed files and split into sub-batches
Format file contents  into a format bulk connectors can parse
Order payments
Publish to Kafka payment topic
Merge back sub-file and sub-batch results into a consolidated response
Bulk connector
Format data based on payment connector requirements
Starting point of initiated payment type-specific workflows
Stream sub-batch transactions from S3 based on metadata from Kafka
Configurations
To provide a granular level of control, the bulk processor supports multiple configurations for different stages like ordering, formatting, splitting etc.
partylookup.enable: used to enable or disable the partylookup stage
approval.enable: used to enable or disable the approval stage
ordering.enable: used to enable or disable the ordering of transactions in a data set based on a field provided.
ordering.field: the field based on which ordering will be done.
formatting.enable: enable or disable the formatting of data set. If formatting is disabled the default formatting standard is used.
formatting.standard: the formatting standard to be used for transactions, possible values are "DEFAULT", "GSMA".
splitting.enable: enable or disable the splitting of the original csv file into sub-batches. Splitting is done on the sub-batch-size field configuration. 
splitting.sub-batch-size: the size of the single sub batch.
mergeback.enable: enable or disable the merging back of the sub batches result.
success-threshold-check.enable: enable or disable the success threshold check, use this configuration to make sure at least x percentage of transaction is successful in a batch.
success-threshold-check.success-rate: a percentage value that will be used while making the threshold check.
acknowledgement.enable: enable or disable the acknowledgement part of the workflow.
Key Considerations:
Kafka provides auto partitioning within topics based on throughput and data
Scalable and fault-tolerant system provided by the multi-cluster setup
Handle backpressure from Zeebe and throttle workflow initiation
Kafka will act as a buffer in this case
Advantageous when Zeebe is overloaded
Due to resource limitations
Or not fast enough scaling
Reduces cascading Denial of Service for other payment connectors.
Provide a visual workflow representation with Zeebe
Stores every state for workflow instances
Timers surviving node failures without interruptions
Using an efficient, high-performance binary protocol (gRPC) for the clients to connect and receive tasks for execution
Retry mechanism on the entire subprocess
But retry is already in place within subprocesses
Unified payment architecture
Easy integration with AMS if payment staging requires it
In case of beneficiary creation during transfer
Using Raft for log replication
Kafka to write ahead log replication as well.
Easy integration of pre and post-transaction workers
Such as bulk notification
References:
​https://forum.camunda.io/t/zeebe-usecase-architecture-question/1168/6​
​https://stage.docs.zeebe.io/reference/grpc.html#error-handling​
​https://docs.camunda.io/docs/product-manuals/zeebe/deployment-guide/operations/backpressure/​
​https://stage.docs.zeebe.io/bpmn-workflows/multi-instance/multi-instance.html​
​https://camunda.com/blog/2019/08/zeebe-horizontal-scalability/​
​https://docs.camunda.io/docs/0.25/product-manuals/zeebe/bpmn-workflows/embedded-subprocesses/embedded-subprocesses/​
​https://docs.camunda.io/docs/0.25/product-manuals/zeebe/bpmn-workflows/event-subprocesses/event-subprocesses​
​https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7​
API Specification
Provide developer friendly consumer APIs
Enable easy integration by abstracting underlying payment type API complexities
Group common fields across various payment APIs and make em mandatory field
All fields in the spec below are mandatory fields
Allow addition of payment type specific fields as optional
​
1. Bulk Transfer
Use below csv file for refernece.
id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle
0,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1
1,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1
2,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1
3,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1
4,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1
5,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1
6,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1
7,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1
8,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2
9,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2
10,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2
11,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2
Use the below request along with above CSV file 
curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=csv' \
--header 'Platform-TenantId: ibank-usa' \
--header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \
--header 'filename: ph-ee-bulk-demo-6.csv
--form 'data=@"ph-ee-bulk-demo-6.csv"'
Use the below request to start a batch only using the filename
curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=raw' \
--header 'Platform-TenantId: ibank-usa' \
--header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \
--header 'filename: ph-ee-bulk-demo-6.csv'
Use the below request to pass transaction data in the body of the request
curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=raw' \
--header 'Platform-TenantId: ibank-usa' \
--header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "creditParty": [
            {
                "key": "msisdn",
                "value": "8837461856"
            }
        ],
        "debitParty": [
            {
                "key": "accountnumber",
                "value": "003001003874120160"
            }
        ],
        "subType": "SLCB",
        "amount": "820.00",
        "currency": "RWF",
        "descriptionText": "Test Payment"
    }
]'
{
    "batch_id": "738bd830-3bd9-4511-bb42-3d0f5798e014",
    "request_id": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
    "status": "queued"
}
2. Bulk Transfer Status
curl --location --request GET 'https://ops-bk.sandbox.fynarfin.io/api/v1/batchtransactions/3112a0ba-0733-4133-ae24-fc3310cb7dfe' \
--header 'Platform-TenantId: ibank-usa' \
--header 'Authorization: Bearer token'
{  
  "batch_id": "UUID",
  "request_id": "UUID",
  "notes": "Bulk transfers",
  "status": "processing",
  "mode": "{payment_mode}",
  "purpose": "refund",
  "total": 1000,
  "successful": 980,
  "failed": 20,
  "file": "S3 link", 
  "created_at": 1545383037,
  "totalAmount": "1050000.00",
  "successfulAmount": "975000.00",
  "failedAmount": "25000.00",
  "pendingAmount": "50000.00",
  "currency": "SLE"
}
Response File
id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle,status,error_code,error_description
0,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1,SUCCESS,,
1,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1,SUCCESS,,
2,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1,SUCCESS,,
3,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1,FAILURE,422,Inactive Account
4,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1,SUCCESS,,
5,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1,SUCCESS,,
6,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1,SUCCESS,,
7,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1,PENDING,,
8,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2,SUCCESS,,
9,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2,SUCCESS,,
10,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2,PENDING,,
11,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2,SUCCESS,,
3. Batch Details
curl --location --request GET 'https://ops-bk.sandbox.fynarfin.io/api/v1/batchtransactions/45e2baca-b087-4d90-8392-da2961f9b9ed/detail?pageNo=1&pageSize=10' \
--header 'Platform-TenantId: ibank-usa' \
--header 'Authorization: Bearer Token'
[
    {
        "id": 568,
        "workflowInstanceKey": 2251799907439003,
        "transactionId": "e5eea064-1445-4d32-bc55-bd9826c779a0",
        "startedAt": 1629130966000,
        "completedAt": 1629130967000,
        "status": "IN_PROGRESS",
        "statusDetail": null,
        "payeeDfspId": "giraffe-bank",
        "payeePartyId": "7854278651",
        "payeePartyIdType": "MSISDN",
        "payeeFee": null,
        "payeeFeeCurrency": null,
        "payeeQuoteCode": null,
        "payerDfspId": "gorilla-bank",
        "payerPartyId": "7543010",
        "payerPartyIdType": "MSISDN",
        "payerFee": null,
        "payerFeeCurrency": null,
        "payerQuoteCode": null,
        "amount": 448,
        "currency": "USD",
        "direction": "OUTGOING",
        "errorInformation": "{\\\"errorInformation\\\":{\\\"errorDescription\\\":\\\"Invalid responseCode 500 for transfer on PAYER side, transactionId: e5eea064-1445-4d32-bc55-bd9826c779a0 Message: {\\\\\\\"timestamp\\\\\\\":1629130966891,\\\\\\\"status\\\\\\\":500,\\\\\\\"error\\\\\\\":\\\\\\\"Internal Server Error\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"path\\\\\\\":\\\\\\\"/fineract-provider/api/v1/interoperation/transfers\\\\\\\"}\\\",\\\"errorCode\\\":\\\"4101\\\"}}",
        "batchId": "c02a14f0-5e7e-44a1-88eb-5584a21e6f28"
    }
]
4. Get Batches
curl --location --request GET 'http://ops-bk.sandbox.fynarfin.io/api/v1/batches?page=2&size=10&sortedBy=requestFile&sortedOrder=asc' \
--header 'Platform-TenantId: ibank-usa'
{
    "content": [
        {
            "id": 57,
            "batchId": "6bdd4e23-9357-481b-b814-b678b146a01b",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661791795354_ph-ee-bulk-demo-6.csv",
            "totalTransactions": 11,
            "total": 12,
            "ongoing": 0,
            "failed": 3,
            "successful": 9,
            "totalAmount": 6432,
            "successfulAmount": 4510,
            "pendingAmount": 0,
            "failedAmount": 1922,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1663238730228_6bdd4e23-9357-481b-b814-b678b146a01b.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813689497,
            "startedAt": 1661791796000,
            "completedAt": 1661791796000
        },
        {
            "id": 29,
            "batchId": "837f0b1a-d06c-4a7b-a6bb-a274e7e3cf43",
            "subBatchId": null,
            "requestId": "1aff3859d4e8-fab3a4d5-0f4f-4e78-b6b5",
            "requestFile": "1661793332698_ph-ee-bulk-demo-6.csv",
            "totalTransactions": 12,
            "ongoing": 0,
            "failed": 12,
            "completed": 0,
            "totalAmount": 3810,
            "ongoingAmount": null,
            "failedAmount": 3810,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/5618434551624_39484844071623_response.csv",
            "resultGeneratedAt": 1661844072000,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813692279,
            "startedAt": 1661793333000,
            "completedAt": 1661793333000
        },
        {
            "id": 33,
            "batchId": "de2b9caf-07f0-43d0-ab98-34e070f52913",
            "subBatchId": null,
            "requestId": "1",
            "requestFile": "1661798671312_1",
            "totalTransactions": 50,
            "ongoing": null,
            "failed": null,
            "completed": 50,
            "totalAmount": 12490,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": 12490,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661873857624_1661843746423_response.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813699494,
            "startedAt": 1661798674000,
            "completedAt": null
        },
        {
            "id": 34,
            "batchId": "eb94687b-5eee-4bc9-ad8a-e89bf0604dc1",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661840850289_ph-ee-bulk-demo-6.csv",
            "totalTransactions": null,
            "ongoing": null,
            "failed": null,
            "completed": null,
            "totalAmount": null,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661849863624_1661983851623_response.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813750207,
            "startedAt": 1661840852000,
            "completedAt": 1661840853000
        },
        {
            "id": 38,
            "batchId": "7f2a7366-3cc1-4293-8cc7-21f862872f7b",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661840967146_ph-ee-bulk-demo-6.csv",
            "totalTransactions": null,
            "ongoing": null,
            "failed": null,
            "completed": null,
            "totalAmount": null,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661883987873_1661844071623_response.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813750554,
            "startedAt": 1661840967000,
            "completedAt": 1661840968000
        },
        {
            "id": 42,
            "batchId": "a7159479-e0d8-4a77-8f92-8b903b3173b7",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661843665396_ph-ee-bulk-demo-6.csv",
            "totalTransactions": 12,
            "ongoing": 0,
            "failed": 12,
            "completed": 0,
            "totalAmount": null,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661844087873_1661844087873_response.csv",
            "resultGeneratedAt": 1661844088000,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813754434,
            "startedAt": 1661843667000,
            "completedAt": 1661843668000
        }
    ],
    "totalElements": 79,
    "totalPages": 8,
    "last": false,
    "numberOfElements": 10,
    "sort": [
        {
            "direction": "ASC",
            "property": "requestFile",
            "ignoreCase": false,
            "nullHandling": "NATIVE",
            "descending": false,
            "ascending": true
        }
    ],
    "first": false,
    "size": 10,
    "number": 2
}
5. Download Template
curl --location --request GET 'https://bulk-connector.sandbox.fynarfin.io/template?types=PROGRAM,MOJALOOP'
Template.csv
id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle
​
Payment Routing
Route enables user to split payments received and transfer the funds to various vendors/beneficiaries.
Sample request body
{
  "amount": 2000,
  "currency": "INR",
  "transfers": [
    {
      "account": "acc_CPRsN1LkFccllA",
      "amount": 1000,
      "currency": "INR",
      "notes": {
        "branch": "Acme Corp Bangalore North",
        "name": "Gaurav Kumar"
      },
    },
    {
      "account": "acc_CNo3jSI8OkFJJJ",
      "amount": 1000,
      "currency": "INR",
      "notes": {
        "branch": "Acme Corp Bangalore South",
        "name": "Saurav Kumar"
      },
    }
  ]
}
​
With Auto Payout
Targeting individuals
Money is transferred directly to bank accounts / mobile money wallets from the sender float account.
No involvement of AMS.
Without Auto Payout
Targeting organisations
Money is transferred to Fineract wallets of the receivers.
Receiver can choose if they want to withdraw the money out of the wallet
Or consider using the money in float/current/checking account for L2 beneficiaries (clear invoices, payroll) seamlessly in one platform (L1 beneficiary amount remains in closed loop)
This provides an advantage to beneficiary orgs to manage their payouts if they have a legacy current/checking account with mostly manual payout processing.
​
Last modified 6mo ago