Compare commits

...

10 Commits

Author SHA1 Message Date
Daeng Deni Mardaeni
d5495d721e feat(webstatement): tambah pemrosesan data ATM Transaction
- Menambahkan job `ProcessAtmTransactionJob` untuk memproses data transaksi ATM dari file CSV.
  - Implementasi pemrosesan file CSV termasuk pembacaan, pemetaan header, dan simpan data ke model.
  - Menyediakan logging untuk pemantauan jumlah data yang diproses dan jumlah error.
  - Menambahkan mekanisme penanganan error pada setiap proses baris dan file.

- Menambahkan model `AtmTransaction`:
  - Mendeklarasikan atribut yang bisa diisi (`fillable`) seperti `transaction_id`, `txn_amount`, dan lainnya.
  - Mendefinisikan tipe data casting untuk beberapa atribut seperti `txn_amount` dalam tipe decimal dan `booking_date` dalam tipe datetime.

- Menambahkan migration `2025_05_21_150736_create_atm_transactions_table` untuk tabel `atm_transactions`:
  - Tabel memiliki kolom seperti `transaction_id`, `txn_amount`, `booking_date`, dan indeks untuk kolom tertentu.
  - Meng-handle struktur kolom untuk mendukung atribut yang diperlukan di model.

- Memperbarui `MigrasiController`:
  - Menambahkan fungsi `ProcessAtmTransaction` untuk menjadwalkan `ProcessAtmTransactionJob`.
  - Memperbaiki pesan response pada beberapa fungsi agar lebih deskriptif dan konsisten.

- Memperbarui pemanggilan fungsi dari `__invoke` di bagian pemrosesan data (`ProcessAtmTransaction`) untuk period tertentu.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-21 22:14:58 +07:00
Daeng Deni Mardaeni
e511025307 feat(webstatement): tambahkan fitur proses data teller
- Menambahkan job baru `ProcessTellerDataJob` untuk memproses data teller dari file CSV.
- Membuat controller method `ProcessTellerData` di `MigrasiController` untuk dispatch job `ProcessTellerDataJob`.
- Menambahkan model baru `Teller` yang merepresentasikan data teller dengan kolom sesuai header map pada CSV.
- Menambahkan migrasi `2025_05_21_144332_create_tellers_table.php` untuk membuat tabel database `tellers` dengan struktur data lengkap.
- Menambahkan logika pada job untuk membaca, memproses, dan menyimpan data CSV ke dalam database.
- Menyediakan mekanisme pemrosesan file CSV termasuk:
  - Validasi jumlah kolom pada CSV jika tidak sesuai dengan header.
  - Membuat dan menghapus file sementara selama pemrosesan.
  - Mapping data CSV ke atribut model `Teller`.
  - Penanganan error saat menyimpan data ke model `Teller`.
  - Logging proses meliputi jumlah data yang diproses dan error selama jalannya job.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-21 22:01:31 +07:00
Daeng Deni Mardaeni
285c2409ea feat(webstatement): tambah fitur pemrosesan data kategori
- Menambahkan `ProcessCategoryDataJob` untuk memproses data kategori dari file CSV yang diambil melalui SFTP.
- Membuat model `Category` dengan atribut-atribut:
  - `id_category`
  - `date_time`
  - `description`
  - `short_name`
  - `system_ind`
  - `record_status`
  - `co_code`
  - `curr_no`
  - `l_db_cr_ind`
  - `category_code`
- Menambahkan endpoint baru `ProcessCategoryData` di `MigrasiController` untuk memanggil job pemrosesan data kategori.
- Menambahkan migrasi untuk membuat tabel `categories` dengan kolom-kolom yang relevan.
- Memperbaiki bug pada `ProcessStmtEntryDataJob` dengan menambahkan validasi khusus untuk menghindari pemrosesan baris header yang tidak valid.
- Menghapus pemanggilan job yang tidak diperlukan di `MigrasiController`.
- Mengupdate logika pemrosesan file untuk memastikan integritas data dalam job kategori.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-21 21:36:45 +07:00
daengdeni
3c061f40f7 feat(webstatement): enhance statement generation and add narrative formatting logic
- Menambahkan logika baru untuk menghasilkan data statement dalam format JSON dan CSV.
- Memuat data transaksi statement menggunakan model `StmtEntry` dengan relasi `ft` dan `transaction`.
- Mengimplementasikan proses mapping data untuk:
  - Penambahan urutan nomor (sequential numbering).
  - Format tanggal transaksi menggunakan properti `booking_date` dan `date_time`.
  - Klasifikasi jenis transaksi (debit atau kredit).
  - Perhitungan running balance.
  - Pembangkitan narrative deskripsi transaksi.
- Menambahkan fungsi `generateNarrative` untuk menghasilkan deskripsi transaksi berdasarkan logika parameter dinamis.
- Menggunakan model `TempStmtNarrFormat` dan `TempStmtNarrParam` untuk format dan parameter narrative.
- Menambahkan fungsi `getFormatNarrative` untuk mem-parsing format narrative dan memasukkan placeholder value sesuai data transaksi.
- Fungsi `getTransaction` ditambahkan sebagai fallback untuk mengambil field secara spesifik dari transaksi.
- Menyediakan opsi data dalam format CSV menggunakan stream response dengan pemisah data berupa pipe (`|`).
- Menambahkan route baru pada `/` untuk mengakses controller `WebstatementController` dan fungsionalitas ini di route `webstatement.index`.
2025-05-21 21:18:40 +07:00
daengdeni
def0b037a8 feat(statement-processing): update StmtEntry model and optimize data processing logic
- Menambahkan relasi baru pada model `StmtEntry`:
  - `ft()` untuk relasi ke model `TempFundsTransfer` berdasarkan kolom `trans_reference`.
  - `transaction()` untuk relasi ke model `TempTransaction` berdasarkan kolom `transaction_code`.

- Memperbarui `ProcessStmtEntryDataJob`:
  - Mengganti penggunaan model `TempStmtEntry` menjadi `StmtEntry`.
  - Mengubah delimiter parsing file CSV dari `/` menjadi `~`.
  - Menggunakan `stmt_entry_id` sebagai kunci utama dalam metode `updateOrCreate`.
  - Menghapus validasi kolom `_id` pada data yang diproses.

Perubahan ini bertujuan untuk menyelaraskan model dan cara proses data agar lebih akurat dan sesuai dengan kebutuhan sistem.
2025-05-21 21:18:04 +07:00
Daeng Deni Mardaeni
39e356e2ff refactor(webstatement): update data processing and migration logic
- Menonaktifkan pemanggilan fungsi-fungsi yang tidak diperlukan untuk data processing di `MigrasiController`:
  - `processTransactionData`, `processStmtNarrFormatData`, `processAccountData`, dan `ProcessCompanyData`.
- Mengubah argumen di beberapa pemanggilan fungsi untuk data processing agar lebih spesifik ke periode tertentu:
  - Mengganti `$periods` menjadi `['20250512']` untuk `processArrangementData`, `processBillDetailData`, `processFundsTransferData`, dan `processStmtEntryData`.
- Memodifikasi logika skipping folder `_parameter` pada `ProcessFtTxnTypeConditionJob` dengan mengomentari proses pengecekan.
- Mengubah delimiter `fgetcsv` pada `ProcessBillDetailDataJob` dari `;` menjadi `~`.
- Menambahkan file migrasi baru untuk mengubah tipe kolom pada tabel `temp_fund_transfer`:
  - Mengubah semua kolom pada tabel menjadi tipe `text` untuk mendukung data yang lebih besar.
  - Menyediakan metode `down` untuk rollback tipe data kembali ke `string`.

Perubahan ini bertujuan untuk meningkatkan efisiensi proses data, mendukung fleksibilitas data lebih besar pada migrasi database, serta membuka jalan untuk refaktor atau penghapusan fungsi yang tidak digunakan.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-21 08:17:47 +07:00
Daeng Deni Mardaeni
fb6fd534d5 feat(webstatement): tambah fitur pemrosesan data "Data Capture"
- Menambahkan fungsi baru `ProcessDataCaptureData` di `MigrasiController` untuk menjadwalkan pemrosesan data "Data Capture".
- Menambahkan job baru `ProcessDataCaptureDataJob`:
  - Mengambil data "Data Capture" dari file CSV melalui SFTP.
  - Memproses dan menyimpan data ke database dengan validasi dan logging.
  - Mendukung operasi pembaruan (update) atau pembuatan (insert) data menggunakan Eloquent `updateOrCreate`.
- Menambahkan model baru `DataCapture`:
  - Memetakan data ke tabel `data_captures` di database.
  - Mendukung properti `fillable` dan `casts` untuk format data yang valid, termasuk konversi nilai decimal dan tanggal.
- Menambahkan migrasi baru `create_data_captures_table` untuk membuat tabel `data_captures`:
  - Mendefinisikan semua kolom sesuai kebutuhan data "Data Capture".
  - Menetapkan tipe data yang relevan seperti decimal, date, dan datetime.
- Memperbarui metode `migrasiAll` di `MigrasiController` untuk memanggil fungsi pemrosesan baru `ProcessDataCaptureData`.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-20 22:15:44 +07:00
Daeng Deni Mardaeni
359cfea905 feat(webstatement): tambahkan proses data company
Menambahkan fungsi baru dan job baru untuk memproses data perusahaan (Company Data) dari file CSV.

Detail perubahan:
- Menambahkan import `ProcessCompanyDataJob` pada `MigrasiController`.
- Menambahkan fungsi `ProcessCompanyData` di `MigrasiController` untuk memanggil job pemrosesan data perusahaan.
- Mengintegrasikan pemrosesan data perusahaan ke dalam alur fungsi `processData()` di `MigrasiController`.
- Membuat job baru `ProcessCompanyDataJob` untuk membaca dan memproses file `ST.COMPANY.csv`.
  - Job memproses file dari SFTP `sftpStatement` berdasarkan folder `periods`.
  - Ditambahkan validasi file, mapping data CSV ke model `Branch`, dan menyimpan atau memperbarui data ke database.
  - Penanganan error untuk pencatatan log pada kasus file tidak ditemukan, kesalahan kolom, atau error saat menyimpan data.
- Melakukan perubahan nama file default untuk job `ProcessFtTxnTypeConditionJob` dari `TXN_TYPE_CONDITION.csv` menjadi `ST.FT.TXN.TYPE.CONDITION.csv`.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-20 22:06:25 +07:00
Daeng Deni Mardaeni
1c8d9a4a8a fix(webstatement): perbaiki duplikasi kode dalam migrasi ft_txn_type_condition
- Menghapus duplikasi `Schema::create` dalam fungsi `up()` di file migrasi.
- Memastikan struktur tabel `ft_txn_type_condition` dihasilkan dengan benar tanpa deklarasi berulang.
- Tidak ada perubahan struktur tabel selain penghapusan kode yang redundan.
- Peningkatan keterbacaan dan pengurangan kemungkinan error karena kode yang berulang.
- Tidak ada dampak pada data atau fungsionalitas yang telah ada.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-20 21:40:44 +07:00
Daeng Deni Mardaeni
24700c5bd8 feat(webstatement): tambahkan proses pengolahan data FtTxnTypeCondition
- Menambahkan method `ProcessFtTxnTypeConditioData` pada `MigrasiController` untuk memproses data FtTxnTypeCondition melalui dispatching job baru.
- Menambahkan job baru `ProcessFtTxnTypeConditionJob` untuk mengelola dan memproses data TxnTypeCondition dari file `TXN_TYPE_CONDITION.csv`:
  - Membaca file CSV melalui SFTP dengan validasi header dan struktur kolom.
  - Menggunakan model `FtTxnTypeCondition` untuk menyimpan atau memperbarui data ke database.
  - Menambahkan logging untuk memberikan informasi dan menangani pengecualian/error selama proses.
- Menambahkan model `FtTxnTypeCondition` untuk representasi data `ft_txn_type_condition` di database:
  - Tabel memiliki atribut seperti `id`, `date_time`, `transaction_type`, `short_descr`, `txn_code_cr`, dan `txn_code_dr`.
  - Mengatur primary key pada atribut `id` dengan tipe string dan non-incremental.
- Menambahkan migrasi untuk tabel `ft_txn_type_condition`:
  - Menyediakan kolom utama yang diperlukan dengan tipe data sesuai kebutuhan, termasuk `softDelete`.
  - Kolom primary key `id` dengan tipe `string`.
- Memodifikasi proses migrasi utama untuk memanggil method `ProcessFtTxnTypeConditioData`.
- Menangani folder `_parameter` sebagai folder yang diabaikan dalam proses.

Signed-off-by: Daeng Deni Mardaeni <ddeni05@gmail.com>
2025-05-20 21:38:27 +07:00
23 changed files with 2080 additions and 18 deletions

View File

@@ -8,12 +8,18 @@ use Illuminate\Support\Facades\Storage;
use Log;
use Modules\Webstatement\Jobs\ProcessAccountDataJob;
use Modules\Webstatement\Jobs\ProcessArrangementDataJob;
use Modules\Webstatement\Jobs\ProcessAtmTransactionJob;
use Modules\Webstatement\Jobs\ProcessBillDetailDataJob;
use Modules\Webstatement\Jobs\ProcessCategoryDataJob;
use Modules\Webstatement\Jobs\ProcessCompanyDataJob;
use Modules\Webstatement\Jobs\ProcessCustomerDataJob;
use Modules\Webstatement\Jobs\ProcessDataCaptureDataJob;
use Modules\Webstatement\Jobs\ProcessFtTxnTypeConditionJob;
use Modules\Webstatement\Jobs\ProcessFundsTransferDataJob;
use Modules\Webstatement\Jobs\ProcessStmtEntryDataJob;
use Modules\Webstatement\Jobs\ProcessStmtNarrFormatDataJob;
use Modules\Webstatement\Jobs\ProcessStmtNarrParamDataJob;
use Modules\Webstatement\Jobs\ProcessTellerDataJob;
use Modules\Webstatement\Jobs\ProcessTransactionDataJob;
class MigrasiController extends Controller
@@ -102,10 +108,65 @@ class MigrasiController extends Controller
}
}
public function ProcessFtTxnTypeConditioData($periods){
try {
ProcessFtTxnTypeConditionJob::dispatch($periods);
return response()->json(['message' => 'FtTxnTypeCondition processing job has been successfully']);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 500);
}
}
public function processStmtEntryData($periods){
try {
ProcessStmtEntryDataJob::dispatch($periods);
return response()->json(['message' => 'Data TempStmtEntry processing job has been successfully']);
return response()->json(['message' => 'Stmt Entry processing job has been successfully']);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 500);
}
}
public function ProcessCompanyData($periods){
try {
ProcessCompanyDataJob::dispatch($periods);
return response()->json(['message' => 'Company processing job has been successfully']);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 500);
}
}
public function ProcessDataCaptureData($periods){
try {
ProcessDataCaptureDataJob::dispatch($periods);
return response()->json(['message' => 'Data Capture processing job has been successfully']);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 500);
}
}
public function ProcessCategoryData($periods){
try {
ProcessCategoryDataJob::dispatch($periods);
return response()->json(['message' => 'Category processing job has been successfully']);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 500);
}
}
public function ProcessTellerData($periods){
try {
ProcessTellerDataJob::dispatch($periods);
return response()->json(['message' => 'Teller processing job has been successfully']);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 500);
}
}
public function ProcessAtmTransaction($periods){
try {
ProcessAtmTransactionJob::dispatch($periods);
return response()->json(['message' => 'AtmTransaction processing job has been successfully']);
} catch (Exception $e) {
return response()->json(['error' => $e->getMessage()], 500);
}
@@ -119,10 +180,10 @@ class MigrasiController extends Controller
// Get all directories (periods) in the SFTP disk
$allDirectories = $disk->directories();
$this->processTransactionData(['_parameter']);
$this->processStmtNarrParamData(['_parameter']);
$this->processStmtNarrFormatData(['_parameter']);
//$this->processTransactionData(['_parameter']);
//$this->processStmtNarrParamData(['_parameter']);
//$this->processStmtNarrFormatData(['_parameter']);
//$this->ProcessFtTxnTypeConditioData(['_parameter']);
// Filter out the _parameter folder
$periods = array_filter($allDirectories, function($dir) {
@@ -138,12 +199,21 @@ class MigrasiController extends Controller
return response()->json(['message' => 'No valid period folders found in SFTP storage'], 404);
}
$this->processCustomerData(['20250519']);
$this->processAccountData(['20250519']);
$this->processArrangementData($periods);
$this->processBillDetailData($periods);
$this->processFundsTransferData($periods);
$this->processStmtEntryData($periods);
$this->ProcessCategoryData($periods);
//$this->processCustomerData($periods);
//$this->processAccountData($periods);
//$this->processStmtEntryData($periods);
//$this->ProcessDataCaptureData($periods);
//$this->processFundsTransferData($periods);
$this->ProcessTellerData($periods);
$this->ProcessAtmTransaction($periods);
//$this->processArrangementData($periods);
//$this->processBillDetailData($periods);
return response()->json(['message' => 'Data processing job has been successfully']);
}

View File

@@ -4,6 +4,10 @@ namespace Modules\Webstatement\Http\Controllers;
use App\Http\Controllers\Controller;
use Illuminate\Http\Request;
use Modules\Webstatement\Models\StmtEntry;
use Modules\Webstatement\Models\TempFundsTransfer;
use Modules\Webstatement\Models\TempStmtNarrFormat;
use Modules\Webstatement\Models\TempStmtNarrParam;
class WebstatementController extends Controller
{
@@ -12,7 +16,164 @@ class WebstatementController extends Controller
*/
public function index()
{
return view('webstatement::index');
$account_number = '1080425781';
$period = '20250512';
$stmt = StmtEntry::with(['ft','transaction'])->where('account_number',$account_number)->where('booking_date',$period)->get();
$saldo = '23984352604';
$runningBalance = (float) $saldo;
// Map the data to transform or format specific fields
$mappedData = $stmt->sortBy(['ACTUAL.DATE', 'REFERENCE.NUMBER'])->map(function ($item, $index) use (&$runningBalance) {
$runningBalance += (float) $item->amount_lcy;
return [
'NO' => 0, // Use $index instead of $item->count()
'TRANSACTION.DATE' => \Carbon\Carbon::createFromFormat('YmdHi', $item->booking_date . substr($item->ft?->date_time ?? '0000000000', 6, 4))->format('d/m/Y H:i'),
'REFERENCE.NUMBER' => $item->trans_reference,
'TRANSACTION.AMOUNT' => $item->amount_lcy,
'TRANSACTION.TYPE' => $item->amount_lcy < 0 ? 'D' : 'C',
'DESCRIPTION' => $this->generateNarative($item),
'END.BALANCE' => $runningBalance,
'ACTUAL.DATE' => \Carbon\Carbon::createFromFormat('ymdHi', $item->ft?->date_time ?? '2505120000')
->format('d/m/Y H:i'),
];
})->values();
// Then apply the sequential numbers
$mappedData = $mappedData->map(function ($item, $index) {
$item['NO'] = $index + 1;
return $item;
});
return response()->json($mappedData);
$csvFileName = $account_number."_".$period.".csv";
$headers = [
"Content-Type" => "text/csv",
"Content-Disposition" => "attachment; filename={$csvFileName}"
];
$callback = function () use ($mappedData) {
$file = fopen('php://output', 'w');
// Write headers without quotes, using pipe separator
fputs($file, implode('|', array_keys($mappedData[0])) . "\n");
// Write data rows without quotes, using pipe separator
foreach ($mappedData as $row) {
fputs($file, implode('|', $row) . "\n");
}
fclose($file);
};
return response()->stream($callback, 200, $headers);
}
function generateNarative($item){
$narr = '';
if($item->transaction->narr_type){
$narr .= $item->transaction->stmt_narr.' ';
$narr .= $this->getFormatNarrative($item->transaction->narr_type,$item);
} else {
$narr .= $item->transaction->stmt_narr.' ';
}
if($item->ft?->recipt_no) {
$narr .= 'Receipt No: ' . $item->ft->recipt_no;
}
return $narr;
}
function getFormatNarrative($narr,$item){
$narrParam = TempStmtNarrParam::where('_id', $narr)->first();
if (!$narrParam) {
return '';
}
$fmt = '';
if($narrParam->_id=='FTIN'){
$fmt = 'FT.IN';
}elseif($narrParam->_id=='FTOUT'){
$fmt = 'FT.IN';
} else {
$fmt = $narrParam->_id;
}
$narrFormat = TempStmtNarrFormat::where('_id', $fmt)->first();
if (!$narrFormat) {
return '';
}
// Get the format string from the database
$formatString = $narrFormat->text_data ?? '';
// Parse the format string
// Split by the separator ']'
$parts = explode(']', $formatString);
$result = '';
foreach ($parts as $index => $part) {
if (empty($part)) {
continue;
}
if ($index === 0) {
// For the first part, take only what's before the '!'
$splitPart = explode('!', $part);
if (count($splitPart) > 0) {
// Remove quotes, backslashes, and other escape characters
$cleanPart = trim($splitPart[0]);
// Remove quotes at the beginning and end
$cleanPart = preg_replace('/^["\'\\\\]+|["\'\\\\]+$/', '', $cleanPart);
// Remove any remaining backslashes
$cleanPart = str_replace('\\', '', $cleanPart);
// Remove any remaining quotes
$cleanPart = str_replace('"', '', $cleanPart);
$result .= $cleanPart;
}
} else {
// For other parts, these are field placeholders
$fieldName = strtolower(str_replace('.', '_', $part));
// Get the corresponding parameter value from narrParam
$paramValue = null;
// Check if the field exists as a property in narrParam
if (property_exists($narrParam, $fieldName)) {
$paramValue = $narrParam->$fieldName;
} elseif (isset($narrParam->$fieldName)) {
$paramValue = $narrParam->$fieldName;
}
// If we found a value, add it to the result
if ($paramValue !== null) {
$result .= $paramValue;
} else {
// If no value found, try to use the original field name as a fallback
if($fieldName!='recipt_no') {
$result .= $this->getTransaction($item->trans_reference, $fieldName).' ';
// $result .= "[$fieldName]";
}
}
}
}
return $result;
}
public function getTransaction($ref, $field){
$trans = TempFundsTransfer::where('ref_no', $ref)->first();
return $trans->$field ?? "";
}
/**

View File

@@ -0,0 +1,235 @@
<?php
namespace Modules\Webstatement\Jobs;
use Exception;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Modules\Webstatement\Models\AtmTransaction;
class ProcessAtmTransactionJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
private const PARAMETER_FOLDER = '_parameter';
// Konstanta untuk nilai-nilai statis
private const FILE_EXTENSION = '.ST.ATM.csv';
private const CSV_DELIMITER = '~';
private const DISK_NAME = 'sftpStatement';
private const HEADER_MAP = [
'id' => 'transaction_id',
'card_acc_id' => 'card_acc_id',
'pan_number' => 'pan_number',
'txn_type' => 'txn_type',
'merchant_id' => 'merchant_id',
'txn_amount' => 'txn_amount',
'booking_date' => 'booking_date',
'trans_ref' => 'trans_ref',
'retrieval_ref_no' => 'retrieval_ref_no',
'stmt_nos' => 'stmt_nos',
'debit_acct_no' => 'debit_acct_no',
'credit_acct_no' => 'credit_acct_no',
'chrg_amount' => 'chrg_amount',
'value_date' => 'value_date',
'stan_no' => 'stan_no',
'trans_status' => 'trans_status',
'proc_code' => 'proc_code'
];
// Pemetaan bidang header ke kolom model
protected array $periods;
/**
* Create a new job instance.
*/
public function __construct(array $periods = [])
{
$this->periods = $periods;
}
/**
* Execute the job.
*/
public function handle(): void
{
try {
set_time_limit(24 * 60 * 60);
if (empty($this->periods)) {
Log::warning('No periods provided for ATM transaction data processing');
return;
}
$stats = $this->processPeriods();
Log::info("ProcessAtmTransactionJob completed. Total processed: {$stats['processed']}, Total errors: {$stats['errors']}");
} catch (Exception $e) {
Log::error("Error in ProcessAtmTransactionJob: " . $e->getMessage());
throw $e;
}
}
/**
* Process all periods and return statistics
*/
private function processPeriods(): array
{
$disk = Storage::disk(self::DISK_NAME);
$processedCount = 0;
$errorCount = 0;
foreach ($this->periods as $period) {
// Skip the parameter folder
if ($period === self::PARAMETER_FOLDER) {
Log::info("Skipping " . self::PARAMETER_FOLDER . " folder");
continue;
}
$result = $this->processPeriodFile($disk, $period);
$processedCount += $result['processed'];
$errorCount += $result['errors'];
}
return [
'processed' => $processedCount,
'errors' => $errorCount
];
}
/**
* Process a single period file
*/
private function processPeriodFile($disk, string $period): array
{
$filename = $period . self::FILE_EXTENSION;
$filePath = "$period/$filename";
$processedCount = 0;
$errorCount = 0;
Log::info("Processing ATM transaction file: $filePath");
if (!$disk->exists($filePath)) {
Log::warning("File not found: $filePath");
return ['processed' => 0, 'errors' => 0];
}
$tempFilePath = $this->createTempFile($disk, $filePath, $filename);
$result = $this->processCSVFile($tempFilePath, $filePath);
$processedCount += $result['processed'];
$errorCount += $result['errors'];
// Clean up the temporary file
if (file_exists($tempFilePath)) {
unlink($tempFilePath);
}
Log::info("Completed processing $filePath. Processed {$result['processed']} records with {$result['errors']} errors.");
return [
'processed' => $processedCount,
'errors' => $errorCount
];
}
/**
* Create a temporary file for processing
*/
private function createTempFile($disk, string $filePath, string $filename): string
{
$tempFilePath = storage_path("app/temp_$filename");
file_put_contents($tempFilePath, $disk->get($filePath));
return $tempFilePath;
}
/**
* Process a CSV file and import data
*/
private function processCSVFile(string $tempFilePath, string $originalFilePath): array
{
$processedCount = 0;
$errorCount = 0;
$handle = fopen($tempFilePath, "r");
if ($handle === false) {
Log::error("Unable to open file: $originalFilePath");
return ['processed' => 0, 'errors' => 0];
}
// Get the headers from the first row
$headerRow = fgetcsv($handle, 0, self::CSV_DELIMITER);
if (!$headerRow) {
fclose($handle);
return ['processed' => 0, 'errors' => 0];
}
$rowCount = 0;
while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
$rowCount++;
if (count($headerRow) !== count($row)) {
Log::warning("Row $rowCount in $originalFilePath has incorrect column count. Expected: " . count($headerRow) . ", Got: " . count($row));
continue;
}
$result = $this->processRow($headerRow, $row, $rowCount, $originalFilePath);
$processedCount += $result['processed'];
$errorCount += $result['errors'];
}
fclose($handle);
return [
'processed' => $processedCount,
'errors' => $errorCount
];
}
/**
* Process a single row from the CSV file
*/
private function processRow(array $headerRow, array $row, int $rowCount, string $filePath): array
{
// Combine the header row with the data row
$rawData = array_combine($headerRow, $row);
// Map the raw data to our model fields
$data = [];
foreach (self::HEADER_MAP as $csvField => $modelField) {
$data[$modelField] = $rawData[$csvField] ?? null;
}
// Skip header row if it was included in the data
if ($data['transaction_id'] === 'id') {
return ['processed' => 0, 'errors' => 0];
}
try {
// Format dates if needed
/*if (!empty($data['booking_date'])) {
$data['booking_date'] = date('Y-m-d H:i:s', strtotime($data['booking_date']));
}
if (!empty($data['value_date'])) {
$data['value_date'] = date('Y-m-d H:i:s', strtotime($data['value_date']));
}*/
// Create or update the record
AtmTransaction::updateOrCreate(
['transaction_id' => $data['transaction_id']],
$data
);
return ['processed' => 1, 'errors' => 0];
} catch (Exception $e) {
Log::error("Error processing row $rowCount in $filePath: " . $e->getMessage());
return ['processed' => 0, 'errors' => 1];
}
}
}

View File

@@ -70,7 +70,7 @@ class ProcessBillDetailDataJob implements ShouldQueue
$headers = (new TempBillDetail())->getFillable();
$rowCount = 0;
while (($row = fgetcsv($handle, 0, ";")) !== false) {
while (($row = fgetcsv($handle, 0, "~")) !== false) {
$rowCount++;
if (count($headers) === count($row)) {

View File

@@ -0,0 +1,138 @@
<?php
namespace Modules\Webstatement\Jobs;
use Exception;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Modules\Webstatement\Models\Category;
class ProcessCategoryDataJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $periods;
/**
* Create a new job instance.
*/
public function __construct(array $periods = [])
{
$this->periods = $periods;
}
/**
* Execute the job.
*/
public function handle()
: void
{
try {
set_time_limit(24 * 60 * 60);
$disk = Storage::disk('sftpStatement');
$processedCount = 0;
$errorCount = 0;
if (empty($this->periods)) {
Log::warning('No periods provided for category data processing');
return;
}
foreach ($this->periods as $period) {
// Skip the _parameter folder
if ($period === '_parameter') {
Log::info("Skipping _parameter folder");
continue;
}
// Construct the filename based on the period folder name
$filename = "$period.ST.CATEGORY.csv";
$filePath = "$period/$filename";
Log::info("Processing category file: $filePath");
if (!$disk->exists($filePath)) {
Log::warning("File not found: $filePath");
continue;
}
// Create a temporary local copy of the file
$tempFilePath = storage_path("app/temp_$filename");
file_put_contents($tempFilePath, $disk->get($filePath));
$handle = fopen($tempFilePath, "r");
if ($handle !== false) {
// Get the headers from the first row
$headerRow = fgetcsv($handle, 0, "~");
// Map the headers to our model fields
$headerMap = [
'id' => 'id_category',
'date_time' => 'date_time',
'description' => 'description',
'short_name' => 'short_name',
'system_ind' => 'system_ind',
'record_status' => 'record_status',
'co_code' => 'co_code',
'curr_no' => 'curr_no',
'l_db_cr_ind' => 'l_db_cr_ind',
'category_code' => 'category_code'
];
$rowCount = 0;
while (($row = fgetcsv($handle, 0, "~")) !== false) {
$rowCount++;
if (count($headerRow) === count($row)) {
// Combine the header row with the data row
$rawData = array_combine($headerRow, $row);
// Map the raw data to our model fields
$data = [];
foreach ($headerMap as $csvField => $modelField) {
$data[$modelField] = $rawData[$csvField] ?? null;
}
try {
// Skip header row if it was included in the data
if ($data['id_category'] !== 'id') {
// Use firstOrNew instead of updateOrCreate
$category = Category::firstOrNew(['id_category' => $data['id_category']]);
$category->fill($data);
$category->save();
$processedCount++;
}
} catch (Exception $e) {
$errorCount++;
Log::error("Error processing Category at row $rowCount in $filePath: " . $e->getMessage());
}
} else {
Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " . count($headerRow) . ", Got: " . count($row));
}
}
fclose($handle);
Log::info("Completed processing $filePath. Processed $processedCount records with $errorCount errors.");
// Clean up the temporary file
unlink($tempFilePath);
} else {
Log::error("Unable to open file: $filePath");
}
}
Log::info("Category data processing completed. Total processed: $processedCount, Total errors: $errorCount");
} catch (Exception $e) {
Log::error('Error in ProcessCategoryDataJob: ' . $e->getMessage());
throw $e;
}
}
}

View File

@@ -0,0 +1,154 @@
<?php
namespace Modules\Webstatement\Jobs;
use Exception;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Modules\Basicdata\Models\Branch;
class ProcessCompanyDataJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $periods;
protected $filename;
/**
* Create a new job instance.
*/
public function __construct(array $periods = [], string $filename = "ST.COMPANY.csv")
{
$this->periods = $periods;
$this->filename = $filename;
}
/**
* Execute the job.
*/
public function handle(): void
{
try {
set_time_limit(24 * 60 * 60);
$disk = Storage::disk('sftpStatement');
$processedCount = 0;
$errorCount = 0;
if (empty($this->periods)) {
Log::warning('No periods provided for company data processing');
return;
}
foreach ($this->periods as $period) {
// Skip the _parameter folder
if ($period === '_parameter') {
Log::info("Skipping _parameter folder");
continue;
}
// Construct the filepath based on the period folder name
$fileName = "$period.$this->filename";
$filePath = "$period/$fileName";
Log::info("Processing company file: $filePath");
if (!$disk->exists($filePath)) {
Log::warning("File not found: $filePath");
continue;
}
// Create a temporary local copy of the file
$tempFilePath = storage_path("app/temp_{$this->filename}");
file_put_contents($tempFilePath, $disk->get($filePath));
$handle = fopen($tempFilePath, "r");
if ($handle !== false) {
// CSV headers from the file
$csvHeaders = [
'id', 'date_time', 'company_code', 'company_name', 'name_address',
'mnemonic', 'customer_company', 'customer_mnemonic', 'company_group',
'curr_no', 'co_code', 'l_vendor_atm', 'l_vendor_cpc'
];
// Field mapping from CSV to Branch model
$fieldMapping = [
'company_code' => 'code',
'company_name' => 'name',
'name_address' => 'address',
'mnemonic' => 'mnemonic',
'customer_company' => 'customer_company',
'customer_mnemonic' => 'customer_mnemonic',
'company_group' => 'company_group',
'curr_no' => 'curr_no',
'co_code' => 'co_code',
'l_vendor_atm' => 'l_vendor_atm',
'l_vendor_cpc' => 'l_vendor_cpc'
];
$rowCount = 0;
while (($row = fgetcsv($handle, 0, "~")) !== false) {
$rowCount++;
// Skip header row if it exists
if ($rowCount === 1 && (strtolower($row[0]) === 'id' || strtolower($row[2]) === 'company_code')) {
continue;
}
if (count($csvHeaders) === count($row)) {
$csvData = array_combine($csvHeaders, $row);
// Map CSV data to Branch model fields
$branchData = [];
foreach ($fieldMapping as $csvField => $modelField) {
if (isset($csvData[$csvField])) {
// Convert string boolean values to actual booleans for boolean fields
if (in_array($modelField, ['l_vendor_atm', 'l_vendor_cpc'])) {
$branchData[$modelField] = filter_var($csvData[$csvField], FILTER_VALIDATE_BOOLEAN);
} else {
$branchData[$modelField] = $csvData[$csvField];
}
}
}
try {
if (!empty($branchData['code'])) {
Branch::updateOrCreate(
['code' => $branchData['code']],
$branchData
);
$processedCount++;
}
} catch (Exception $e) {
$errorCount++;
Log::error("Error processing Company data at row $rowCount in $filePath: " . $e->getMessage());
}
} else {
Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " . count($csvHeaders) . ", Got: " . count($row));
}
}
fclose($handle);
Log::info("Completed processing $filePath. Processed $processedCount records with $errorCount errors.");
// Clean up the temporary file
unlink($tempFilePath);
} else {
Log::error("Unable to open file: $filePath");
}
}
Log::info("Company data processing completed. Total processed: $processedCount, Total errors: $errorCount");
} catch (Exception $e) {
Log::error('Error in ProcessCompanyDataJob: ' . $e->getMessage());
throw $e;
}
}
}

View File

@@ -0,0 +1,185 @@
<?php
namespace Modules\Webstatement\Jobs;
use Exception;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Modules\Webstatement\Models\DataCapture;
class ProcessDataCaptureDataJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $periods;
protected $filename;
/**
* Create a new job instance.
*/
public function __construct(array $periods = [], string $filename = "ST.DATA.CAPTURE.csv")
{
$this->periods = $periods;
$this->filename = $filename;
}
/**
* Execute the job.
*/
public function handle()
: void
{
try {
set_time_limit(24 * 60 * 60);
$disk = Storage::disk('sftpStatement');
$processedCount = 0;
$errorCount = 0;
if (empty($this->periods)) {
Log::warning('No periods provided for data capture processing');
return;
}
foreach ($this->periods as $period) {
// Skip the _parameter folder
if ($period === '_parameter') {
Log::info("Skipping _parameter folder");
continue;
}
// Construct the filepath based on the period folder name
$fileName = "$period.$this->filename";
$filePath = "$period/$fileName";
Log::info("Processing data capture file: $filePath");
if (!$disk->exists($filePath)) {
Log::warning("File not found: $filePath");
continue;
}
// Create a temporary local copy of the file
$tempFilePath = storage_path("app/temp_{$this->filename}");
file_put_contents($tempFilePath, $disk->get($filePath));
$handle = fopen($tempFilePath, "r");
if ($handle !== false) {
// CSV headers from the file
$csvHeaders = [
'id',
'account_number',
'sign',
'amount_lcy',
'transaction_code',
'their_reference',
'narrative',
'pl_category',
'customer_id',
'account_officer',
'product_category',
'value_date',
'currency',
'amount_fcy',
'exchange_rate',
'neg_ref_no',
'position_type',
'our_reference',
'reversal_marker',
'exposure_date',
'currency_market',
'iblc_country',
'last_version',
'otor_version',
'department_code',
'dealer_desk',
'bank_sort_cde',
'cheque_number',
'accounting_date',
'contingent_acct',
'cheq_type',
'tfs_reference',
'accounting_company',
'stmt_no',
'curr_no',
'inputter',
'authoriser',
'co_code',
'date_time'
];
$rowCount = 0;
while (($row = fgetcsv($handle, 0, "~")) !== false) {
$rowCount++;
// Skip header row if it exists
if ($rowCount === 1 && strtolower($row[0]) === 'id') {
continue;
}
if (count($csvHeaders) === count($row)) {
$data = array_combine($csvHeaders, $row);
try {
// Format dates if they exist
foreach (['value_date', 'exposure_date', 'accounting_date'] as $dateField) {
if (!empty($data[$dateField])) {
try {
$data[$dateField] = date('Y-m-d', strtotime($data[$dateField]));
} catch (Exception $e) {
// If date parsing fails, keep the original value
Log::warning("Failed to parse date for $dateField: {$data[$dateField]}");
}
}
}
// Format datetime if it exists
if (!empty($data['date_time'])) {
try {
$data['date_time'] = date('Y-m-d H:i:s', strtotime($data['date_time']));
} catch (Exception $e) {
// If datetime parsing fails, keep the original value
Log::warning("Failed to parse datetime for date_time: {$data['date_time']}");
}
}
if (!empty($data['id'])) {
DataCapture::updateOrCreate(
['id' => $data['id']],
$data
);
$processedCount++;
}
} catch (Exception $e) {
$errorCount++;
Log::error("Error processing Data Capture at row $rowCount in $filePath: " . $e->getMessage());
}
} else {
Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " . count($csvHeaders) . ", Got: " . count($row));
}
}
fclose($handle);
Log::info("Completed processing $filePath. Processed $processedCount records with $errorCount errors.");
// Clean up the temporary file
unlink($tempFilePath);
} else {
Log::error("Unable to open file: $filePath");
}
}
Log::info("Data capture processing completed. Total processed: $processedCount, Total errors: $errorCount");
} catch (Exception $e) {
Log::error('Error in ProcessDataCaptureDataJob: ' . $e->getMessage());
throw $e;
}
}
}

View File

@@ -0,0 +1,118 @@
<?php
namespace Modules\Webstatement\Jobs;
use Exception;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Modules\Webstatement\Models\FtTxnTypeCondition;
class ProcessFtTxnTypeConditionJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $periods;
protected $filename;
/**
* Create a new job instance.
*/
public function __construct(array $periods = [], string $filename = "ST.FT.TXN.TYPE.CONDITION.csv")
{
$this->periods = $periods;
$this->filename = $filename;
}
/**
* Execute the job.
*/
public function handle(): void
{
try {
set_time_limit(24 * 60 * 60);
$disk = Storage::disk('sftpStatement');
$processedCount = 0;
$errorCount = 0;
if (empty($this->periods)) {
Log::warning('No periods provided for transaction type condition data processing');
return;
}
foreach ($this->periods as $period) {
// Skip the _parameter folder
/*if ($period === '_parameter') {
Log::info("Skipping _parameter folder");
continue;
}*/
// Construct the filepath based on the period folder name
$filePath = "$period/{$this->filename}";
Log::info("Processing transaction type condition file: $filePath");
if (!$disk->exists($filePath)) {
Log::warning("File not found: $filePath");
continue;
}
// Create a temporary local copy of the file
$tempFilePath = storage_path("app/temp_{$this->filename}");
file_put_contents($tempFilePath, $disk->get($filePath));
$handle = fopen($tempFilePath, "r");
if ($handle !== false) {
$headers = ['id', 'date_time', 'transaction_type', 'short_descr', 'txn_code_cr', 'txn_code_dr'];
$rowCount = 0;
while (($row = fgetcsv($handle, 0, "~")) !== false) {
$rowCount++;
// Skip header row if it exists
if ($rowCount === 1 && strtolower($row[0]) === 'id') {
continue;
}
if (count($headers) === count($row)) {
$data = array_combine($headers, $row);
try {
if (!empty($data['id'])) {
FtTxnTypeCondition::updateOrCreate(
['id' => $data['id']],
$data
);
$processedCount++;
}
} catch (Exception $e) {
$errorCount++;
Log::error("Error processing Transaction Type Condition at row $rowCount in $filePath: " . $e->getMessage());
}
} else {
Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " . count($headers) . ", Got: " . count($row));
}
}
fclose($handle);
Log::info("Completed processing $filePath. Processed $processedCount records with $errorCount errors.");
// Clean up the temporary file
unlink($tempFilePath);
} else {
Log::error("Unable to open file: $filePath");
}
}
Log::info("Transaction type condition data processing completed. Total processed: $processedCount, Total errors: $errorCount");
} catch (Exception $e) {
Log::error('Error in ProcessFtTxnTypeConditionJob: ' . $e->getMessage());
throw $e;
}
}
}

View File

@@ -10,6 +10,7 @@
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Modules\Webstatement\Models\StmtEntry;
use Modules\Webstatement\Models\TempStmtEntry;
class ProcessStmtEntryDataJob implements ShouldQueue
@@ -67,18 +68,18 @@
$handle = fopen($tempFilePath, "r");
if ($handle !== false) {
$headers = (new TempStmtEntry())->getFillable();
$headers = (new StmtEntry())->getFillable();
$rowCount = 0;
while (($row = fgetcsv($handle, 0, "/")) !== false) {
while (($row = fgetcsv($handle, 0, "~")) !== false) {
$rowCount++;
if (count($headers) === count($row)) {
$data = array_combine($headers, $row);
try {
if (isset($data['_id']) && $data['_id'] !== '_id') {
TempStmtEntry::updateOrCreate(
['_id' => $data['_id']],
if ($data['stmt_entry_id'] !== 'stmt_entry_id') {
StmtEntry::updateOrCreate(
['stmt_entry_id' => $data['stmt_entry_id']],
$data
);
$processedCount++;

View File

@@ -0,0 +1,316 @@
<?php
namespace Modules\Webstatement\Jobs;
use Exception;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Modules\Webstatement\Models\Teller;
class ProcessTellerDataJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
private const PARAMETER_FOLDER = '_parameter';
// Konstanta untuk nilai-nilai statis
private const FILE_EXTENSION = '.ST.TELLER.csv';
private const CSV_DELIMITER = '~';
private const DISK_NAME = 'sftpStatement';
private const HEADER_MAP = [
'id' => 'id_teller',
'account_1' => 'account_1',
'currency_1' => 'currency_1',
'amount_local_1' => 'amount_local_1',
'value_date_1' => 'value_date_1',
'account_2' => 'account_2',
'new_cust_bal' => 'new_cust_bal',
'term_type' => 'term_type',
'term_id' => 'term_id',
'trans_reff' => 'trans_reff',
'card_no' => 'card_no',
'recipt_no' => 'recipt_no',
'transaction_code' => 'transaction_code',
'date_time' => 'date_time',
'record_status' => 'record_status',
'amount_local_2' => 'amount_local_2',
'co_code' => 'co_code',
'narrative_1' => 'narrative_1',
'wic_flag' => 'wic_flag',
'wic_cust_type' => 'wic_cust_type',
'wic_full_name' => 'wic_full_name',
'wic_alias_name' => 'wic_alias_name',
'wic_acct_no' => 'wic_acct_no',
'wic_id_type' => 'wic_id_type',
'wic_id_no' => 'wic_id_no',
'wic_npwp' => 'wic_npwp',
'wic_nationality' => 'wic_nationality',
'wic_ind_birthpl' => 'wic_ind_birthpl',
'wic_ind_birthdt' => 'wic_ind_birthdt',
'wic_address_id' => 'wic_address_id',
'wic_address_cur' => 'wic_address_cur',
'wic_city' => 'wic_city',
'wic_province' => 'wic_province',
'wic_post_code' => 'wic_post_code',
'wic_phone' => 'wic_phone',
'wic_gender' => 'wic_gender',
'wic_marital_sts' => 'wic_marital_sts',
'wic_occptn' => 'wic_occptn',
'wic_occptn_dur' => 'wic_occptn_dur',
'wic_income_avg' => 'wic_income_avg',
'wic_cor_name' => 'wic_cor_name',
'wic_cor_address' => 'wic_cor_address',
'wic_cor_phone' => 'wic_cor_phone',
'wic_cor_lgl_typ' => 'wic_cor_lgl_typ',
'wic_cor_lic_no' => 'wic_cor_lic_no',
'wic_cor_birthpl' => 'wic_cor_birthpl',
'wic_cor_birthdt' => 'wic_cor_birthdt',
'wic_cor_rel' => 'wic_cor_rel',
'wic_party_rel' => 'wic_party_rel',
'wic_amount' => 'wic_amount',
'wic_amount_type' => 'wic_amount_type',
'wic_amtbnk_name' => 'wic_amtbnk_name',
'wic_amtbnk_cunm' => 'wic_amtbnk_cunm',
'wic_fund_source' => 'wic_fund_source',
'wic_fund_use' => 'wic_fund_use',
'dr_cr_marker' => 'dr_cr_marker',
'charge_code' => 'charge_code',
'chrg_amt_local' => 'chrg_amt_local',
'charge_category' => 'charge_category',
'charge_account' => 'charge_account',
'amount_fcy_1' => 'amount_fcy_1',
'rate_1' => 'rate_1',
'deal_rate' => 'deal_rate',
'l_wic_id' => 'l_wic_id',
'account_1_co_code' => 'account_1_co_code',
'account_2_co_code' => 'account_2_co_code',
'l_charge_amt' => 'l_charge_amt',
'bl_cust_no' => 'bl_cust_no',
'stmt_no' => 'stmt_no',
'bil_customer' => 'bil_customer',
'value_date_2' => 'value_date_2',
'cheq_type' => 'cheq_type',
'cheque_number' => 'cheque_number',
'inputter' => 'inputter',
'authoriser' => 'authoriser',
'bil_product' => 'bil_product',
'fx_document' => 'fx_document',
'fx_purpose' => 'fx_purpose',
'narrative_2' => 'narrative_2',
'customer_2' => 'customer_2',
'l_sms_1' => 'l_sms_1',
'l_phone_1' => 'l_phone_1',
'kyc_incom_rng' => 'kyc_incom_rng',
'wic_rt' => 'wic_rt',
'wic_rw' => 'wic_rw',
'ktp_kelurahan' => 'ktp_kelurahan',
'ktp_kecamatan' => 'ktp_kecamatan',
'ktp_provinsi' => 'ktp_provinsi',
'wic_jenis_kelam' => 'wic_jenis_kelam',
'kyc_sumber_dana' => 'kyc_sumber_dana',
'l_manual_risk' => 'l_manual_risk',
'l_slip_no' => 'l_slip_no',
'dest_bank_id' => 'dest_bank_id',
'dr_narrative' => 'dr_narrative',
'inter_type' => 'inter_type',
'l_va_number' => 'l_va_number',
'inter_bank_id' => 'inter_bank_id',
'term_narr' => 'term_narr',
'currency_2' => 'currency_2',
'amount_fcy_2' => 'amount_fcy_2',
'rate_2' => 'rate_2',
'customer_1' => 'customer_1',
'last_version' => 'last_version'
];
// Pemetaan bidang header ke kolom model
protected array $periods;
/**
* Create a new job instance.
*/
public function __construct(array $periods = [])
{
$this->periods = $periods;
}
/**
* Execute the job.
*/
public function handle()
: void
{
try {
set_time_limit(24 * 60 * 60);
if (empty($this->periods)) {
Log::warning('No periods provided for teller data processing');
return;
}
$stats = $this->processPeriods();
Log::info("ProcessTellerDataJob completed. Total processed: {$stats['processed']}, Total errors: {$stats['errors']}");
} catch (Exception $e) {
Log::error("Error in ProcessTellerDataJob: " . $e->getMessage());
throw $e;
}
}
/**
* Process all periods and return statistics
*/
private function processPeriods()
: array
{
$disk = Storage::disk(self::DISK_NAME);
$processedCount = 0;
$errorCount = 0;
foreach ($this->periods as $period) {
// Skip the parameter folder
if ($period === self::PARAMETER_FOLDER) {
Log::info("Skipping " . self::PARAMETER_FOLDER . " folder");
continue;
}
$result = $this->processPeriodFile($disk, $period);
$processedCount += $result['processed'];
$errorCount += $result['errors'];
}
return [
'processed' => $processedCount,
'errors' => $errorCount
];
}
/**
* Process a single period file
*/
private function processPeriodFile($disk, string $period)
: array
{
$filename = $period . self::FILE_EXTENSION;
$filePath = "$period/$filename";
$processedCount = 0;
$errorCount = 0;
Log::info("Processing teller file: $filePath");
if (!$disk->exists($filePath)) {
Log::warning("File not found: $filePath");
return ['processed' => 0, 'errors' => 0];
}
$tempFilePath = $this->createTempFile($disk, $filePath, $filename);
$result = $this->processCSVFile($tempFilePath, $filePath);
$processedCount += $result['processed'];
$errorCount += $result['errors'];
// Clean up the temporary file
if (file_exists($tempFilePath)) {
unlink($tempFilePath);
}
Log::info("Completed processing $filePath. Processed {$result['processed']} records with {$result['errors']} errors.");
return [
'processed' => $processedCount,
'errors' => $errorCount
];
}
/**
* Create a temporary file for processing
*/
private function createTempFile($disk, string $filePath, string $filename)
: string
{
$tempFilePath = storage_path("app/temp_$filename");
file_put_contents($tempFilePath, $disk->get($filePath));
return $tempFilePath;
}
/**
* Process a CSV file and import data
*/
private function processCSVFile(string $tempFilePath, string $originalFilePath)
: array
{
$processedCount = 0;
$errorCount = 0;
$handle = fopen($tempFilePath, "r");
if ($handle === false) {
Log::error("Unable to open file: $originalFilePath");
return ['processed' => 0, 'errors' => 0];
}
// Get the headers from the first row
$headerRow = fgetcsv($handle, 0, self::CSV_DELIMITER);
if (!$headerRow) {
fclose($handle);
return ['processed' => 0, 'errors' => 0];
}
$rowCount = 0;
while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
$rowCount++;
if (count($headerRow) !== count($row)) {
Log::warning("Row $rowCount in $originalFilePath has incorrect column count. Expected: " . count($headerRow) . ", Got: " . count($row));
continue;
}
$result = $this->processRow($headerRow, $row, $rowCount, $originalFilePath);
$processedCount += $result['processed'];
$errorCount += $result['errors'];
}
fclose($handle);
return [
'processed' => $processedCount,
'errors' => $errorCount
];
}
/**
* Process a single row from the CSV file
*/
private function processRow(array $headerRow, array $row, int $rowCount, string $filePath)
: array
{
// Combine the header row with the data row
$rawData = array_combine($headerRow, $row);
// Map the raw data to our model fields
$data = [];
foreach (self::HEADER_MAP as $csvField => $modelField) {
$data[$modelField] = $rawData[$csvField] ?? null;
}
// Skip header row if it was included in the data
if ($data['id_teller'] === 'id') {
return ['processed' => 0, 'errors' => 0];
}
try {
$teller = Teller::firstOrNew(['id_teller' => $data['id_teller']]);
$teller->fill($data);
$teller->save();
return ['processed' => 1, 'errors' => 0];
} catch (Exception $e) {
Log::error("Error processing Teller at row $rowCount in $filePath: " . $e->getMessage());
return ['processed' => 0, 'errors' => 1];
}
}
}

View File

@@ -0,0 +1,45 @@
<?php
namespace Modules\Webstatement\Models;
use Illuminate\Database\Eloquent\Model;
class AtmTransaction extends Model
{
/**
* The attributes that are mass assignable.
*
* @var array<int, string>
*/
protected $fillable = [
'transaction_id',
'card_acc_id',
'pan_number',
'txn_type',
'merchant_id',
'txn_amount',
'booking_date',
'trans_ref',
'retrieval_ref_no',
'stmt_nos',
'debit_acct_no',
'credit_acct_no',
'chrg_amount',
'value_date',
'stan_no',
'trans_status',
'proc_code',
];
/**
* The attributes that should be cast.
*
* @var array<string, string>
*/
protected $casts = [
'booking_date' => 'datetime',
'value_date' => 'datetime',
'txn_amount' => 'decimal:2',
'chrg_amount' => 'decimal:2',
];
}

24
app/Models/Category.php Normal file
View File

@@ -0,0 +1,24 @@
<?php
namespace Modules\Webstatement\Models;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Factories\HasFactory;
class Category extends Model
{
use HasFactory;
protected $fillable = [
'id_category',
'date_time',
'description',
'short_name',
'system_ind',
'record_status',
'co_code',
'curr_no',
'l_db_cr_ind',
'category_code'
];
}

View File

@@ -0,0 +1,74 @@
<?php
namespace Modules\Webstatement\Models;
use Illuminate\Database\Eloquent\Model;
class DataCapture extends Model
{
/**
* Indicates if the model's ID is auto-incrementing.
*
* @var bool
*/
public $incrementing = false;
protected $table = 'data_captures';
/**
* The data type of the auto-incrementing ID.
*
* @var string
*/
protected $keyType = 'string';
protected $fillable = [
'id',
'account_number',
'sign',
'amount_lcy',
'transaction_code',
'their_reference',
'narrative',
'pl_category',
'customer_id',
'account_officer',
'product_category',
'value_date',
'currency',
'amount_fcy',
'exchange_rate',
'neg_ref_no',
'position_type',
'our_reference',
'reversal_marker',
'exposure_date',
'currency_market',
'iblc_country',
'last_version',
'otor_version',
'department_code',
'dealer_desk',
'bank_sort_cde',
'cheque_number',
'accounting_date',
'contingent_acct',
'cheq_type',
'tfs_reference',
'accounting_company',
'stmt_no',
'curr_no',
'inputter',
'authoriser',
'co_code',
'date_time'
];
protected $casts = [
'amount_lcy' => 'decimal:2',
'amount_fcy' => 'decimal:2',
'exchange_rate' => 'decimal:6',
'value_date' => 'date',
'exposure_date' => 'date',
'accounting_date' => 'date',
'date_time' => 'datetime'
];
}

View File

@@ -0,0 +1,61 @@
<?php
namespace Modules\Webstatement\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
// use Modules\Webstatement\Database\Factories\FtTxnTypeConditionFactory;
class FtTxnTypeCondition extends Model
{
use HasFactory;
/**
* Indicates if the IDs are auto-incrementing.
*
* @var bool
*/
public $incrementing = false;
/**
* The table associated with the model.
*
* @var string
*/
protected $table = 'ft_txn_type_condition';
/**
* The primary key for the model.
*
* @var string
*/
protected $primaryKey = 'id';
/**
* The data type of the auto-incrementing ID.
*
* @var string
*/
protected $keyType = 'string';
/**
* The attributes that are mass assignable.
*
* @var array<int, string>
*/
protected $fillable = [
'id',
'date_time',
'transaction_type',
'short_descr',
'txn_code_cr',
'txn_code_dr',
];
/**
* The attributes that should be cast.
*
* @var array<string, string>
*/
protected $casts = [
'date_time' => 'datetime',
];
}

View File

@@ -56,4 +56,12 @@ class StmtEntry extends Model
{
return $this->belongsTo(Account::class, 'account_number', 'account_number');
}
public function ft(){
return $this->belongsTo(TempFundsTransfer::class, 'trans_reference', 'ref_no');
}
public function transaction(){
return $this->belongsTo(TempTransaction::class, 'transaction_code', 'transaction_code');
}
}

117
app/Models/Teller.php Normal file
View File

@@ -0,0 +1,117 @@
<?php
namespace Modules\Webstatement\Models;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Factories\HasFactory;
class Teller extends Model
{
use HasFactory;
protected $fillable = [
'id_teller',
'account_1',
'currency_1',
'amount_local_1',
'value_date_1',
'account_2',
'new_cust_bal',
'term_type',
'term_id',
'trans_reff',
'card_no',
'recipt_no',
'transaction_code',
'date_time',
'record_status',
'amount_local_2',
'co_code',
'narrative_1',
'wic_flag',
'wic_cust_type',
'wic_full_name',
'wic_alias_name',
'wic_acct_no',
'wic_id_type',
'wic_id_no',
'wic_npwp',
'wic_nationality',
'wic_ind_birthpl',
'wic_ind_birthdt',
'wic_address_id',
'wic_address_cur',
'wic_city',
'wic_province',
'wic_post_code',
'wic_phone',
'wic_gender',
'wic_marital_sts',
'wic_occptn',
'wic_occptn_dur',
'wic_income_avg',
'wic_cor_name',
'wic_cor_address',
'wic_cor_phone',
'wic_cor_lgl_typ',
'wic_cor_lic_no',
'wic_cor_birthpl',
'wic_cor_birthdt',
'wic_cor_rel',
'wic_party_rel',
'wic_amount',
'wic_amount_type',
'wic_amtbnk_name',
'wic_amtbnk_cunm',
'wic_fund_source',
'wic_fund_use',
'dr_cr_marker',
'charge_code',
'chrg_amt_local',
'charge_category',
'charge_account',
'amount_fcy_1',
'rate_1',
'deal_rate',
'l_wic_id',
'account_1_co_code',
'account_2_co_code',
'l_charge_amt',
'bl_cust_no',
'stmt_no',
'bil_customer',
'value_date_2',
'cheq_type',
'cheque_number',
'inputter',
'authoriser',
'bil_product',
'fx_document',
'fx_purpose',
'narrative_2',
'customer_2',
'l_sms_1',
'l_phone_1',
'kyc_incom_rng',
'wic_rt',
'wic_rw',
'ktp_kelurahan',
'ktp_kecamatan',
'ktp_provinsi',
'wic_jenis_kelam',
'kyc_sumber_dana',
'l_manual_risk',
'l_slip_no',
'dest_bank_id',
'dr_narrative',
'inter_type',
'l_va_number',
'inter_bank_id',
'term_narr',
'currency_2',
'amount_fcy_2',
'rate_2',
'customer_1',
'last_version'
];
}

View File

@@ -0,0 +1,33 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('ft_txn_type_condition', function (Blueprint $table) {
$table->string('id')->primary();
$table->dateTime('date_time')->nullable();
$table->string('transaction_type')->nullable();
$table->string('short_descr')->nullable();
$table->string('txn_code_cr')->nullable();
$table->string('txn_code_dr')->nullable();
$table->timestamps();
$table->softDeletes();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('ft_txn_type_condition');
}
};

View File

@@ -0,0 +1,65 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('data_captures', function (Blueprint $table) {
$table->string('id')->primary();
$table->string('account_number')->nullable();
$table->string('sign')->nullable();
$table->decimal('amount_lcy', 20, 2)->nullable();
$table->string('transaction_code')->nullable();
$table->string('their_reference')->nullable();
$table->text('narrative')->nullable();
$table->string('pl_category')->nullable();
$table->string('customer_id')->nullable();
$table->string('account_officer')->nullable();
$table->string('product_category')->nullable();
$table->date('value_date')->nullable();
$table->string('currency')->nullable();
$table->decimal('amount_fcy', 20, 2)->nullable();
$table->decimal('exchange_rate', 20, 6)->nullable();
$table->string('neg_ref_no')->nullable();
$table->string('position_type')->nullable();
$table->string('our_reference')->nullable();
$table->string('reversal_marker')->nullable();
$table->date('exposure_date')->nullable();
$table->string('currency_market')->nullable();
$table->string('iblc_country')->nullable();
$table->string('last_version')->nullable();
$table->string('otor_version')->nullable();
$table->string('department_code')->nullable();
$table->string('dealer_desk')->nullable();
$table->string('bank_sort_cde')->nullable();
$table->string('cheque_number')->nullable();
$table->date('accounting_date')->nullable();
$table->string('contingent_acct')->nullable();
$table->string('cheq_type')->nullable();
$table->string('tfs_reference')->nullable();
$table->string('accounting_company')->nullable();
$table->string('stmt_no')->nullable();
$table->string('curr_no')->nullable();
$table->string('inputter')->nullable();
$table->string('authoriser')->nullable();
$table->string('co_code')->nullable();
$table->dateTime('date_time')->nullable();
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('data_captures');
}
};

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,37 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('categories', function (Blueprint $table) {
$table->id();
$table->string('id_category')->nullable();
$table->string('date_time')->nullable();
$table->text('description')->nullable();
$table->string('short_name')->nullable();
$table->string('system_ind')->nullable();
$table->string('record_status')->nullable();
$table->string('co_code')->nullable();
$table->string('curr_no')->nullable();
$table->string('l_db_cr_ind')->nullable();
$table->string('category_code')->nullable();
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('categories');
}
};

View File

@@ -0,0 +1,130 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('tellers', function (Blueprint $table) {
$table->id();
$table->string('id_teller')->nullable();
$table->string('account_1')->nullable();
$table->string('currency_1')->nullable();
$table->string('amount_local_1')->nullable();
$table->string('value_date_1')->nullable();
$table->string('account_2')->nullable();
$table->string('new_cust_bal')->nullable();
$table->string('term_type')->nullable();
$table->string('term_id')->nullable();
$table->string('trans_reff')->nullable();
$table->string('card_no')->nullable();
$table->string('recipt_no')->nullable();
$table->string('transaction_code')->nullable();
$table->string('date_time')->nullable();
$table->string('record_status')->nullable();
$table->string('amount_local_2')->nullable();
$table->string('co_code')->nullable();
$table->text('narrative_1')->nullable();
$table->string('wic_flag')->nullable();
$table->string('wic_cust_type')->nullable();
$table->string('wic_full_name')->nullable();
$table->string('wic_alias_name')->nullable();
$table->string('wic_acct_no')->nullable();
$table->string('wic_id_type')->nullable();
$table->string('wic_id_no')->nullable();
$table->string('wic_npwp')->nullable();
$table->string('wic_nationality')->nullable();
$table->string('wic_ind_birthpl')->nullable();
$table->string('wic_ind_birthdt')->nullable();
$table->string('wic_address_id')->nullable();
$table->string('wic_address_cur')->nullable();
$table->string('wic_city')->nullable();
$table->string('wic_province')->nullable();
$table->string('wic_post_code')->nullable();
$table->string('wic_phone')->nullable();
$table->string('wic_gender')->nullable();
$table->string('wic_marital_sts')->nullable();
$table->string('wic_occptn')->nullable();
$table->string('wic_occptn_dur')->nullable();
$table->string('wic_income_avg')->nullable();
$table->string('wic_cor_name')->nullable();
$table->string('wic_cor_address')->nullable();
$table->string('wic_cor_phone')->nullable();
$table->string('wic_cor_lgl_typ')->nullable();
$table->string('wic_cor_lic_no')->nullable();
$table->string('wic_cor_birthpl')->nullable();
$table->string('wic_cor_birthdt')->nullable();
$table->string('wic_cor_rel')->nullable();
$table->string('wic_party_rel')->nullable();
$table->string('wic_amount')->nullable();
$table->string('wic_amount_type')->nullable();
$table->string('wic_amtbnk_name')->nullable();
$table->string('wic_amtbnk_cunm')->nullable();
$table->string('wic_fund_source')->nullable();
$table->string('wic_fund_use')->nullable();
$table->string('dr_cr_marker')->nullable();
$table->string('charge_code')->nullable();
$table->string('chrg_amt_local')->nullable();
$table->string('charge_category')->nullable();
$table->string('charge_account')->nullable();
$table->string('amount_fcy_1')->nullable();
$table->string('rate_1')->nullable();
$table->string('deal_rate')->nullable();
$table->string('l_wic_id')->nullable();
$table->string('account_1_co_code')->nullable();
$table->string('account_2_co_code')->nullable();
$table->string('l_charge_amt')->nullable();
$table->string('bl_cust_no')->nullable();
$table->string('stmt_no')->nullable();
$table->string('bil_customer')->nullable();
$table->string('value_date_2')->nullable();
$table->string('cheq_type')->nullable();
$table->string('cheque_number')->nullable();
$table->string('inputter')->nullable();
$table->string('authoriser')->nullable();
$table->string('bil_product')->nullable();
$table->string('fx_document')->nullable();
$table->string('fx_purpose')->nullable();
$table->text('narrative_2')->nullable();
$table->string('customer_2')->nullable();
$table->string('l_sms_1')->nullable();
$table->string('l_phone_1')->nullable();
$table->string('kyc_incom_rng')->nullable();
$table->string('wic_rt')->nullable();
$table->string('wic_rw')->nullable();
$table->string('ktp_kelurahan')->nullable();
$table->string('ktp_kecamatan')->nullable();
$table->string('ktp_provinsi')->nullable();
$table->string('wic_jenis_kelam')->nullable();
$table->string('kyc_sumber_dana')->nullable();
$table->string('l_manual_risk')->nullable();
$table->string('l_slip_no')->nullable();
$table->string('dest_bank_id')->nullable();
$table->text('dr_narrative')->nullable();
$table->string('inter_type')->nullable();
$table->string('l_va_number')->nullable();
$table->string('inter_bank_id')->nullable();
$table->text('term_narr')->nullable();
$table->string('currency_2')->nullable();
$table->string('amount_fcy_2')->nullable();
$table->string('rate_2')->nullable();
$table->string('customer_1')->nullable();
$table->string('last_version')->nullable();
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('tellers');
}
};

View File

@@ -0,0 +1,45 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration {
/**
* Run the migrations.
*/
public function up()
: void
{
Schema::create('atm_transactions', function (Blueprint $table) {
$table->id();
$table->string('transaction_id')->nullable()->index();
$table->string('card_acc_id')->nullable();
$table->string('pan_number')->nullable();
$table->string('txn_type')->nullable();
$table->string('merchant_id')->nullable();
$table->string('txn_amount')->nullable();
$table->string('booking_date')->nullable();
$table->string('trans_ref')->nullable();
$table->string('retrieval_ref_no')->nullable();
$table->string('stmt_nos')->nullable();
$table->string('debit_acct_no')->nullable();
$table->string('credit_acct_no')->nullable();
$table->string('chrg_amount')->nullable();
$table->string('value_date')->nullable();
$table->string('stan_no')->nullable();
$table->string('trans_status')->nullable();
$table->string('proc_code')->nullable();
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down()
: void
{
Schema::dropIfExists('atm_transactions');
}
};

View File

@@ -93,3 +93,4 @@ Route::get('migrasi', [MigrasiController::class, 'index'])->name('migrasi.index'
Route::get('biaya-kartu', [SyncLogsController::class, 'index'])->name('biaya-kartu.index');
Route::get('/stmt-entries/{accountNumber}', [MigrasiController::class, 'getStmtEntryByAccount']);
Route::get('/', [WebstatementController::class, 'index'])->name('webstatement.index');