You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
56 KiB
56 KiB
| 1 | agent_count | test_name | model_name | latency_ms | throughput_rps | memory_usage_mb | cpu_usage_percent | success_rate | error_count | total_requests | concurrent_requests | timestamp | cost_usd | tokens_used | response_quality_score | additional_metrics | agent_creation_time | tool_registration_time | execution_time | total_latency | chaining_steps | chaining_success | error_scenarios_tested | recovery_rate | resource_cycles | avg_memory_delta | memory_leak_detected |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 1 | scaling_test | gpt-4o-mini | 1131.7063331604004 | 4.131429224630576 | 1.25 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345643.9453266 | 0.0015359999999999996 | 10240 | 0.8548663728748707 | {'min_latency_ms': 562.7951622009277, 'max_latency_ms': 1780.4391384124756, 'p95_latency_ms': np.float64(1744.0685987472534), 'p99_latency_ms': np.float64(1773.1650304794312), 'total_time_s': 4.84093976020813, 'initial_memory_mb': 291.5546875, 'final_memory_mb': 292.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.679999999999998e-05, 'quality_std': 0.0675424923987846, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 3 | 6 | scaling_test | gpt-4o-mini | 1175.6950378417969 | 3.7575854004826277 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345654.225195 | 0.0015359999999999996 | 10240 | 0.8563524483655013 | {'min_latency_ms': 535.4223251342773, 'max_latency_ms': 1985.3930473327637, 'p95_latency_ms': np.float64(1975.6355285644531), 'p99_latency_ms': np.float64(1983.4415435791016), 'total_time_s': 5.322566986083984, 'initial_memory_mb': 293.1796875, 'final_memory_mb': 293.1796875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.679999999999998e-05, 'quality_std': 0.05770982402152013, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 4 | 11 | scaling_test | gpt-4o-mini | 996.9684720039368 | 4.496099509029146 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345662.8977199 | 0.0015359999999999996 | 10240 | 0.8844883644941982 | {'min_latency_ms': 45.22204399108887, 'max_latency_ms': 1962.2983932495117, 'p95_latency_ms': np.float64(1647.7753758430483), 'p99_latency_ms': np.float64(1899.3937897682185), 'total_time_s': 4.448300123214722, 'initial_memory_mb': 293.5546875, 'final_memory_mb': 293.5546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.679999999999998e-05, 'quality_std': 0.043434832388308614, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 5 | 16 | scaling_test | gpt-4o-mini | 1112.8681421279907 | 3.587833950074127 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345673.162652 | 0.0015359999999999996 | 10240 | 0.8563855623109009 | {'min_latency_ms': 564.1369819641113, 'max_latency_ms': 1951.472282409668, 'p95_latency_ms': np.float64(1897.4883794784546), 'p99_latency_ms': np.float64(1940.6755018234253), 'total_time_s': 5.57439398765564, 'initial_memory_mb': 293.8046875, 'final_memory_mb': 293.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.679999999999998e-05, 'quality_std': 0.05691925404970228, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 6 | 1 | scaling_test | gpt-4o | 1298.2240080833435 | 3.3670995599405846 | 0.125 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345683.2065425 | 0.0512 | 10240 | 0.9279627852934385 | {'min_latency_ms': 693.6078071594238, 'max_latency_ms': 1764.8026943206787, 'p95_latency_ms': np.float64(1681.7602753639221), 'p99_latency_ms': np.float64(1748.1942105293274), 'total_time_s': 5.939830303192139, 'initial_memory_mb': 293.8046875, 'final_memory_mb': 293.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00256, 'quality_std': 0.050879141399088765, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 7 | 6 | scaling_test | gpt-4o | 1264.4854545593262 | 3.5293826102318846 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345692.6439528 | 0.0512 | 10240 | 0.9737471278894755 | {'min_latency_ms': 175.65083503723145, 'max_latency_ms': 1990.2207851409912, 'p95_latency_ms': np.float64(1910.3824019432068), 'p99_latency_ms': np.float64(1974.2531085014343), 'total_time_s': 5.66671347618103, 'initial_memory_mb': 293.9296875, 'final_memory_mb': 293.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00256, 'quality_std': 0.038542680129780495, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 8 | 11 | scaling_test | gpt-4o | 1212.0607376098633 | 3.799000004302323 | 0.125 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345701.8719423 | 0.0512 | 10240 | 0.9366077507029601 | {'min_latency_ms': 542.8001880645752, 'max_latency_ms': 1973.801851272583, 'p95_latency_ms': np.float64(1969.2555904388428), 'p99_latency_ms': np.float64(1972.892599105835), 'total_time_s': 5.264543294906616, 'initial_memory_mb': 293.9296875, 'final_memory_mb': 294.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00256, 'quality_std': 0.044670864578792276, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 9 | 16 | scaling_test | gpt-4o | 1367.1631932258606 | 3.1229790107314654 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345711.9738443 | 0.0512 | 10240 | 0.9328922198254587 | {'min_latency_ms': 715.888261795044, 'max_latency_ms': 1905.6315422058105, 'p95_latency_ms': np.float64(1890.480661392212), 'p99_latency_ms': np.float64(1902.6013660430908), 'total_time_s': 6.404141664505005, 'initial_memory_mb': 294.0546875, 'final_memory_mb': 294.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00256, 'quality_std': 0.05146728864962903, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 10 | 1 | scaling_test | gpt-4-turbo | 1429.1370868682861 | 3.3141614744089267 | 0.125 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345722.7650242 | 0.1024 | 10240 | 0.960928099222926 | {'min_latency_ms': 637.6686096191406, 'max_latency_ms': 1994.9300289154053, 'p95_latency_ms': np.float64(1973.6997246742249), 'p99_latency_ms': np.float64(1990.6839680671692), 'total_time_s': 6.0347089767456055, 'initial_memory_mb': 294.0546875, 'final_memory_mb': 294.1796875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00512, 'quality_std': 0.0429193742204114, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 11 | 6 | scaling_test | gpt-4-turbo | 1167.8012132644653 | 3.933946564951724 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345731.809648 | 0.1024 | 10240 | 0.9575695597206497 | {'min_latency_ms': 521.2328433990479, 'max_latency_ms': 1973.503828048706, 'p95_latency_ms': np.float64(1931.3542008399963), 'p99_latency_ms': np.float64(1965.073902606964), 'total_time_s': 5.083953142166138, 'initial_memory_mb': 294.1796875, 'final_memory_mb': 294.1796875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00512, 'quality_std': 0.04742414087184447, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 12 | 11 | scaling_test | gpt-4-turbo | 1435.1954460144043 | 3.0793869953124613 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345741.9117725 | 0.1024 | 10240 | 0.9564233524947511 | {'min_latency_ms': 711.4903926849365, 'max_latency_ms': 2034.2109203338623, 'p95_latency_ms': np.float64(1998.979663848877), 'p99_latency_ms': np.float64(2027.1646690368652), 'total_time_s': 6.4947991371154785, 'initial_memory_mb': 294.3046875, 'final_memory_mb': 294.3046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00512, 'quality_std': 0.03428874308764032, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 13 | 16 | scaling_test | gpt-4-turbo | 1092.1013355255127 | 4.057819053252887 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345749.8833907 | 0.1024 | 10240 | 0.9521218582720758 | {'min_latency_ms': 554.4416904449463, 'max_latency_ms': 1968.658447265625, 'p95_latency_ms': np.float64(1637.098050117493), 'p99_latency_ms': np.float64(1902.346367835998), 'total_time_s': 4.92875599861145, 'initial_memory_mb': 294.3046875, 'final_memory_mb': 294.3046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00512, 'quality_std': 0.043763298033728824, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 14 | 1 | scaling_test | claude-3-5-sonnet | 1046.9236850738525 | 4.047496446876068 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345757.9539518 | 0.03071999999999999 | 10240 | 0.9511838758969231 | {'min_latency_ms': 184.94415283203125, 'max_latency_ms': 1966.0136699676514, 'p95_latency_ms': np.float64(1677.8094530105593), 'p99_latency_ms': np.float64(1908.3728265762325), 'total_time_s': 4.941326141357422, 'initial_memory_mb': 294.3046875, 'final_memory_mb': 294.3046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0015359999999999996, 'quality_std': 0.03727295215254124, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 15 | 6 | scaling_test | claude-3-5-sonnet | 1381.3772201538086 | 3.283979343278356 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345768.7153368 | 0.03071999999999999 | 10240 | 0.957817098536435 | {'min_latency_ms': 543.0643558502197, 'max_latency_ms': 1937.4654293060303, 'p95_latency_ms': np.float64(1931.4598441123962), 'p99_latency_ms': np.float64(1936.2643122673035), 'total_time_s': 6.090172290802002, 'initial_memory_mb': 294.3046875, 'final_memory_mb': 294.3046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0015359999999999996, 'quality_std': 0.044335695599357156, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 16 | 11 | scaling_test | claude-3-5-sonnet | 1314.3961310386658 | 3.5243521468336656 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345778.6269403 | 0.03071999999999999 | 10240 | 0.9749641888502683 | {'min_latency_ms': 535.1722240447998, 'max_latency_ms': 1983.6831092834473, 'p95_latency_ms': np.float64(1918.512487411499), 'p99_latency_ms': np.float64(1970.6489849090576), 'total_time_s': 5.674801826477051, 'initial_memory_mb': 294.3046875, 'final_memory_mb': 294.3046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0015359999999999996, 'quality_std': 0.03856740540886548, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 17 | 16 | scaling_test | claude-3-5-sonnet | 1120.720875263214 | 3.7028070875807546 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345788.3161702 | 0.03071999999999999 | 10240 | 0.9344569749738585 | {'min_latency_ms': 207.9324722290039, 'max_latency_ms': 2018.561601638794, 'p95_latency_ms': np.float64(1963.4979844093323), 'p99_latency_ms': np.float64(2007.5488781929016), 'total_time_s': 5.401307582855225, 'initial_memory_mb': 294.3046875, 'final_memory_mb': 294.3046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0015359999999999996, 'quality_std': 0.04750434388073592, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 18 | 1 | scaling_test | claude-3-haiku | 1268.5401320457458 | 3.539921687652236 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345797.6495905 | 0.0256 | 10240 | 0.8406194607723803 | {'min_latency_ms': 534.9514484405518, 'max_latency_ms': 1956.9103717803955, 'p95_latency_ms': np.float64(1938.3319020271301), 'p99_latency_ms': np.float64(1953.1946778297424), 'total_time_s': 5.6498425006866455, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00128, 'quality_std': 0.053962632063170944, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 19 | 6 | scaling_test | claude-3-haiku | 1377.644693851471 | 3.189212271479164 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345808.2179801 | 0.0256 | 10240 | 0.8370154862115219 | {'min_latency_ms': 661.4456176757812, 'max_latency_ms': 2013.9634609222412, 'p95_latency_ms': np.float64(1985.2455973625183), 'p99_latency_ms': np.float64(2008.2198882102966), 'total_time_s': 6.271141052246094, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00128, 'quality_std': 0.057589803133820325, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 20 | 11 | scaling_test | claude-3-haiku | 1161.9974493980408 | 3.6778795132801156 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345817.2541294 | 0.0256 | 10240 | 0.8421329247896683 | {'min_latency_ms': 549.6580600738525, 'max_latency_ms': 1785.23588180542, 'p95_latency_ms': np.float64(1730.9520959854126), 'p99_latency_ms': np.float64(1774.3791246414185), 'total_time_s': 5.437916040420532, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00128, 'quality_std': 0.05774508247670216, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 21 | 16 | scaling_test | claude-3-haiku | 1365.4750227928162 | 2.998821435629251 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345827.8750126 | 0.0256 | 10240 | 0.8483772503724578 | {'min_latency_ms': 767.146110534668, 'max_latency_ms': 1936.8767738342285, 'p95_latency_ms': np.float64(1919.3583130836487), 'p99_latency_ms': np.float64(1933.3730816841125), 'total_time_s': 6.669286727905273, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00128, 'quality_std': 0.05705131022796498, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 22 | 1 | scaling_test | claude-3-sonnet | 1360.187566280365 | 3.089520735450049 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345837.7737727 | 0.15360000000000001 | 10240 | 0.8835217044830507 | {'min_latency_ms': 550.3547191619873, 'max_latency_ms': 1977.1480560302734, 'p95_latency_ms': np.float64(1924.659264087677), 'p99_latency_ms': np.float64(1966.6502976417542), 'total_time_s': 6.473495960235596, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000001, 'quality_std': 0.058452629496046606, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 23 | 6 | scaling_test | claude-3-sonnet | 1256.138801574707 | 3.4732685564079335 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345848.5701082 | 0.15360000000000001 | 10240 | 0.8863139635356961 | {'min_latency_ms': 641.2796974182129, 'max_latency_ms': 1980.7326793670654, 'p95_latency_ms': np.float64(1846.4025855064392), 'p99_latency_ms': np.float64(1953.86666059494), 'total_time_s': 5.758264780044556, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000001, 'quality_std': 0.05783521510861833, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 24 | 11 | scaling_test | claude-3-sonnet | 1306.07008934021 | 3.5020347317551495 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345858.6472163 | 0.15360000000000001 | 10240 | 0.9094961422561505 | {'min_latency_ms': 591.8083190917969, 'max_latency_ms': 1971.1270332336426, 'p95_latency_ms': np.float64(1944.3620324134827), 'p99_latency_ms': np.float64(1965.7740330696106), 'total_time_s': 5.710965633392334, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000001, 'quality_std': 0.042442911768923584, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 25 | 16 | scaling_test | claude-3-sonnet | 1307.1481943130493 | 3.262938882676132 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345869.905544 | 0.15360000000000001 | 10240 | 0.8938240662052681 | {'min_latency_ms': 646.7251777648926, 'max_latency_ms': 1990.9627437591553, 'p95_latency_ms': np.float64(1935.0676536560059), 'p99_latency_ms': np.float64(1979.7837257385254), 'total_time_s': 6.129443645477295, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000001, 'quality_std': 0.04247877605865338, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 26 | 1 | scaling_test | gemini-1.5-pro | 1401.3476371765137 | 2.943218490521141 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345881.238218 | 0.0128 | 10240 | 0.9409363720199192 | {'min_latency_ms': 520.9827423095703, 'max_latency_ms': 1970.2589511871338, 'p95_latency_ms': np.float64(1958.1118822097778), 'p99_latency_ms': np.float64(1967.8295373916626), 'total_time_s': 6.7952821254730225, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00064, 'quality_std': 0.05267230653872383, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 27 | 6 | scaling_test | gemini-1.5-pro | 1341.485834121704 | 3.3982951582179024 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345889.5553467 | 0.0128 | 10240 | 0.9355344625586725 | {'min_latency_ms': 503.9515495300293, 'max_latency_ms': 1978.0657291412354, 'p95_latency_ms': np.float64(1966.320013999939), 'p99_latency_ms': np.float64(1975.716586112976), 'total_time_s': 5.885303974151611, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00064, 'quality_std': 0.054780000845711954, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 28 | 11 | scaling_test | gemini-1.5-pro | 1344.3536400794983 | 3.445457146125384 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345898.4512925 | 0.0128 | 10240 | 0.9276983017835836 | {'min_latency_ms': 615.3252124786377, 'max_latency_ms': 1981.612205505371, 'p95_latency_ms': np.float64(1803.935217857361), 'p99_latency_ms': np.float64(1946.0768079757688), 'total_time_s': 5.8047449588775635, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00064, 'quality_std': 0.05905363250623063, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 29 | 16 | scaling_test | gemini-1.5-pro | 1202.2199511528015 | 3.696869831400932 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345907.5707264 | 0.0128 | 10240 | 0.9307740387961949 | {'min_latency_ms': 589.9953842163086, 'max_latency_ms': 1967.3075675964355, 'p95_latency_ms': np.float64(1913.6008977890015), 'p99_latency_ms': np.float64(1956.5662336349487), 'total_time_s': 5.409982204437256, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00064, 'quality_std': 0.04978369465928124, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 30 | 1 | scaling_test | gemini-1.5-flash | 1053.9512276649475 | 3.823265280376166 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345915.0947819 | 0.007679999999999998 | 10240 | 0.8813998853517441 | {'min_latency_ms': -36.76271438598633, 'max_latency_ms': 1967.0710563659668, 'p95_latency_ms': np.float64(1855.4362535476685), 'p99_latency_ms': np.float64(1944.744095802307), 'total_time_s': 5.231130599975586, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0003839999999999999, 'quality_std': 0.050008698196664016, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 31 | 6 | scaling_test | gemini-1.5-flash | 1155.3911447525024 | 3.615636866719992 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345925.0694563 | 0.007679999999999998 | 10240 | 0.9025102091839412 | {'min_latency_ms': 502.6116371154785, 'max_latency_ms': 1947.0453262329102, 'p95_latency_ms': np.float64(1765.414369106293), 'p99_latency_ms': np.float64(1910.7191348075864), 'total_time_s': 5.531528949737549, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0003839999999999999, 'quality_std': 0.059194105459554974, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 32 | 11 | scaling_test | gemini-1.5-flash | 1217.6612257957458 | 3.756965086673101 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345934.1183383 | 0.007679999999999998 | 10240 | 0.8709830012564668 | {'min_latency_ms': 560.8868598937988, 'max_latency_ms': 2007.932424545288, 'p95_latency_ms': np.float64(1776.0017752647402), 'p99_latency_ms': np.float64(1961.5462946891782), 'total_time_s': 5.323445796966553, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0003839999999999999, 'quality_std': 0.052873446152615404, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 33 | 16 | scaling_test | gemini-1.5-flash | 1351.5228390693665 | 3.367995990496259 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345942.2099788 | 0.007679999999999998 | 10240 | 0.872315613940513 | {'min_latency_ms': 689.1014575958252, 'max_latency_ms': 1980.147361755371, 'p95_latency_ms': np.float64(1956.2964797019958), 'p99_latency_ms': np.float64(1975.377185344696), 'total_time_s': 5.938249349594116, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0003839999999999999, 'quality_std': 0.05361394744479093, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 34 | 1 | scaling_test | llama-3.1-8b | 1306.591236591339 | 3.3070039261320594 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345952.8692935 | 0.002048000000000001 | 10240 | 0.7778348786353027 | {'min_latency_ms': 555.4070472717285, 'max_latency_ms': 1988.0244731903076, 'p95_latency_ms': np.float64(1957.3988199234009), 'p99_latency_ms': np.float64(1981.8993425369263), 'total_time_s': 6.047770261764526, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010240000000000006, 'quality_std': 0.05832225784189981, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 35 | 6 | scaling_test | llama-3.1-8b | 1199.6222853660583 | 3.634358086220239 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345963.5152647 | 0.002048000000000001 | 10240 | 0.7696592403957419 | {'min_latency_ms': 541.0621166229248, 'max_latency_ms': 1914.41011428833, 'p95_latency_ms': np.float64(1768.0468797683716), 'p99_latency_ms': np.float64(1885.1374673843382), 'total_time_s': 5.503035068511963, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010240000000000006, 'quality_std': 0.06176209698043544, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 36 | 11 | scaling_test | llama-3.1-8b | 1143.358552455902 | 4.173916297150752 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345973.8406181 | 0.002048000000000001 | 10240 | 0.7857043630038748 | {'min_latency_ms': 631.817102432251, 'max_latency_ms': 1720.1111316680908, 'p95_latency_ms': np.float64(1547.544610500336), 'p99_latency_ms': np.float64(1685.5978274345396), 'total_time_s': 4.791662931442261, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010240000000000006, 'quality_std': 0.06142254552174686, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 37 | 16 | scaling_test | llama-3.1-8b | 1228.6048531532288 | 3.613465135130269 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345982.2759545 | 0.002048000000000001 | 10240 | 0.7706622409066766 | {'min_latency_ms': 539.0913486480713, 'max_latency_ms': 1971.7633724212646, 'p95_latency_ms': np.float64(1819.2362308502197), 'p99_latency_ms': np.float64(1941.2579441070554), 'total_time_s': 5.534853458404541, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010240000000000006, 'quality_std': 0.05320944570994387, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 38 | 1 | scaling_test | llama-3.1-70b | 1424.0724563598633 | 2.989394263900763 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759345993.4949126 | 0.008192000000000005 | 10240 | 0.8731561293258354 | {'min_latency_ms': 700.6974220275879, 'max_latency_ms': 1959.3937397003174, 'p95_latency_ms': np.float64(1924.493396282196), 'p99_latency_ms': np.float64(1952.4136710166931), 'total_time_s': 6.690318584442139, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00040960000000000025, 'quality_std': 0.0352234743129485, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 39 | 6 | scaling_test | llama-3.1-70b | 1090.003514289856 | 4.145917207566353 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759346002.3353932 | 0.008192000000000005 | 10240 | 0.8796527768140011 | {'min_latency_ms': 508.23211669921875, 'max_latency_ms': 1798.6392974853516, 'p95_latency_ms': np.float64(1785.5579257011414), 'p99_latency_ms': np.float64(1796.0230231285095), 'total_time_s': 4.824023008346558, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00040960000000000025, 'quality_std': 0.06407982743031454, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 40 | 11 | scaling_test | llama-3.1-70b | 964.3666982650757 | 4.70392645090585 | 0.0 | 0.0 | 1.0 | 0 | 20 | 5 | 1759346010.6974216 | 0.008192000000000005 | 10240 | 0.8992009479579495 | {'min_latency_ms': 135.56504249572754, 'max_latency_ms': 1794.3906784057617, 'p95_latency_ms': np.float64(1775.5030393600464), 'p99_latency_ms': np.float64(1790.6131505966187), 'total_time_s': 4.251767158508301, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.4296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00040960000000000025, 'quality_std': 0.050182727925105516, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 41 | 16 | scaling_test | llama-3.1-70b | 1258.9476823806763 | 3.653831604110515 | 0.125 | 0.0 | 1.0 | 0 | 20 | 5 | 1759346020.388094 | 0.008192000000000005 | 10240 | 0.8930892849911802 | {'min_latency_ms': 620.0413703918457, 'max_latency_ms': 1916.384220123291, 'p95_latency_ms': np.float64(1765.2448296546936), 'p99_latency_ms': np.float64(1886.1563420295713), 'total_time_s': 5.473706007003784, 'initial_memory_mb': 294.4296875, 'final_memory_mb': 294.5546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00040960000000000025, 'quality_std': 0.04969618373257882, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 42 | 5 | concurrent_test | gpt-4o-mini | 1273.702096939087 | 0.7851086796926611 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346033.2373884 | 0.0007680000000000001 | 5120 | 0.8342026655690804 | {'min_latency_ms': 741.3482666015625, 'max_latency_ms': 1817.1906471252441, 'p95_latency_ms': np.float64(1794.5520520210266), 'p99_latency_ms': np.float64(1812.6629281044006), 'total_time_s': 12.737090110778809, 'initial_memory_mb': 294.5546875, 'final_memory_mb': 294.5546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.680000000000001e-05, 'quality_std': 0.0446055902590032, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 43 | 5 | concurrent_test | gpt-4o-mini | 1511.399483680725 | 2.933763102440156 | 0.25 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346036.647214 | 0.0007680000000000001 | 5120 | 0.8471277213854321 | {'min_latency_ms': 800.0023365020752, 'max_latency_ms': 1982.2335243225098, 'p95_latency_ms': np.float64(1942.5656914710999), 'p99_latency_ms': np.float64(1974.2999577522278), 'total_time_s': 3.4085915088653564, 'initial_memory_mb': 294.5546875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.680000000000001e-05, 'quality_std': 0.06432848764341552, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 44 | 5 | concurrent_test | gpt-4o | 1150.0491619110107 | 0.8695228900132853 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346048.2587333 | 0.0256 | 5120 | 0.9599583095352598 | {'min_latency_ms': 544.191837310791, 'max_latency_ms': 1584.9177837371826, 'p95_latency_ms': np.float64(1511.2051010131834), 'p99_latency_ms': np.float64(1570.1752471923828), 'total_time_s': 11.50055980682373, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00256, 'quality_std': 0.057087428808928614, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 45 | 5 | concurrent_test | gpt-4o | 1241.9081926345825 | 3.22981029743519 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346051.3563757 | 0.0256 | 5120 | 0.9585199558650109 | {'min_latency_ms': 644.8915004730225, 'max_latency_ms': 1933.1202507019043, 'p95_latency_ms': np.float64(1865.2720570564268), 'p99_latency_ms': np.float64(1919.5506119728088), 'total_time_s': 3.0961570739746094, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00256, 'quality_std': 0.04062204558012218, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 46 | 5 | concurrent_test | gpt-4-turbo | 1581.8750381469727 | 0.6321581179029606 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346067.3017964 | 0.0512 | 5120 | 0.9324427514695872 | {'min_latency_ms': 833.935022354126, 'max_latency_ms': 2019.5622444152832, 'p95_latency_ms': np.float64(1978.4671545028687), 'p99_latency_ms': np.float64(2011.3432264328003), 'total_time_s': 15.818827152252197, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00512, 'quality_std': 0.04654046504268862, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 47 | 5 | concurrent_test | gpt-4-turbo | 1153.432297706604 | 3.2168993240245847 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346070.4116762 | 0.0512 | 5120 | 0.9790878168553954 | {'min_latency_ms': 635.2591514587402, 'max_latency_ms': 1833.7628841400146, 'p95_latency_ms': np.float64(1808.298635482788), 'p99_latency_ms': np.float64(1828.6700344085693), 'total_time_s': 3.108583450317383, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00512, 'quality_std': 0.038783270511690816, 'data_size_processed': 1000, 'model_provider': 'gpt'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 48 | 5 | concurrent_test | claude-3-5-sonnet | 1397.6783752441406 | 0.7154680102707422 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346084.5017824 | 0.015359999999999999 | 5120 | 0.9421283071854264 | {'min_latency_ms': 532.8092575073242, 'max_latency_ms': 2028.5301208496094, 'p95_latency_ms': np.float64(1968.815779685974), 'p99_latency_ms': np.float64(2016.5872526168823), 'total_time_s': 13.976865291595459, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0015359999999999998, 'quality_std': 0.041911119259679885, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 49 | 5 | concurrent_test | claude-3-5-sonnet | 1215.26198387146 | 3.6278421983995233 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346087.2596216 | 0.015359999999999999 | 5120 | 0.9131170426955485 | {'min_latency_ms': 568.2053565979004, 'max_latency_ms': 1612.9648685455322, 'p95_latency_ms': np.float64(1559.6276402473447), 'p99_latency_ms': np.float64(1602.2974228858948), 'total_time_s': 2.7564594745635986, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0015359999999999998, 'quality_std': 0.04319876804321411, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 50 | 5 | concurrent_test | claude-3-haiku | 1299.2276906967163 | 0.7696826190331395 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346100.364407 | 0.0128 | 5120 | 0.8252745814485088 | {'min_latency_ms': 668.3671474456787, 'max_latency_ms': 2041.351318359375, 'p95_latency_ms': np.float64(1843.0875778198238), 'p99_latency_ms': np.float64(2001.6985702514648), 'total_time_s': 12.992368221282959, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00128, 'quality_std': 0.058205855327116265, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 51 | 5 | concurrent_test | claude-3-haiku | 1297.508192062378 | 3.6581654644321087 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346103.0993996 | 0.0128 | 5120 | 0.8496515913760503 | {'min_latency_ms': 649.4293212890625, 'max_latency_ms': 1873.1675148010254, 'p95_latency_ms': np.float64(1843.8988208770752), 'p99_latency_ms': np.float64(1867.3137760162354), 'total_time_s': 2.7336106300354004, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00128, 'quality_std': 0.06872259975771335, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 52 | 5 | concurrent_test | claude-3-sonnet | 1239.8123741149902 | 0.8065692205263874 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346114.9650035 | 0.07680000000000001 | 5120 | 0.8917269647002374 | {'min_latency_ms': 559.9334239959717, 'max_latency_ms': 1828.9196491241455, 'p95_latency_ms': np.float64(1804.089903831482), 'p99_latency_ms': np.float64(1823.9537000656128), 'total_time_s': 12.398191928863525, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000001, 'quality_std': 0.06728256480558785, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 53 | 5 | concurrent_test | claude-3-sonnet | 1325.3875255584717 | 3.2305613290400945 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346118.062173 | 0.07680000000000001 | 5120 | 0.8904253939966993 | {'min_latency_ms': 598.4294414520264, 'max_latency_ms': 1956.3815593719482, 'p95_latency_ms': np.float64(1906.8223834037778), 'p99_latency_ms': np.float64(1946.4697241783142), 'total_time_s': 3.0954372882843018, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000001, 'quality_std': 0.06220445402424322, 'data_size_processed': 1000, 'model_provider': 'claude'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 54 | 5 | concurrent_test | gemini-1.5-pro | 1264.2754554748535 | 0.7909630217832475 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346130.8282964 | 0.0064 | 5120 | 0.8998460053229075 | {'min_latency_ms': 532.9890251159668, 'max_latency_ms': 1795.492172241211, 'p95_latency_ms': np.float64(1745.6329107284544), 'p99_latency_ms': np.float64(1785.5203199386597), 'total_time_s': 12.642816066741943, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00064, 'quality_std': 0.04050886994282564, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 55 | 5 | concurrent_test | gemini-1.5-pro | 1342.9006338119507 | 3.7829150181123015 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346133.472956 | 0.0064 | 5120 | 0.9029938738274873 | {'min_latency_ms': 701.9498348236084, 'max_latency_ms': 1964.576005935669, 'p95_latency_ms': np.float64(1872.5560665130613), 'p99_latency_ms': np.float64(1946.1720180511475), 'total_time_s': 2.6434640884399414, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00064, 'quality_std': 0.05723923041822323, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 56 | 5 | concurrent_test | gemini-1.5-flash | 1368.2588577270508 | 0.7308515907093506 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346147.2717574 | 0.0038399999999999997 | 5120 | 0.8795901650694117 | {'min_latency_ms': 620.3913688659668, 'max_latency_ms': 2018.2685852050781, 'p95_latency_ms': np.float64(1993.7742233276367), 'p99_latency_ms': np.float64(2013.3697128295898), 'total_time_s': 13.682668447494507, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00038399999999999996, 'quality_std': 0.05927449072307118, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 57 | 5 | concurrent_test | gemini-1.5-flash | 1207.8629732131958 | 3.2879592824302044 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346150.314617 | 0.0038399999999999997 | 5120 | 0.8611774574826484 | {'min_latency_ms': 594.973087310791, 'max_latency_ms': 1811.2657070159912, 'p95_latency_ms': np.float64(1681.6352963447569), 'p99_latency_ms': np.float64(1785.3396248817444), 'total_time_s': 3.041400194168091, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00038399999999999996, 'quality_std': 0.07904328865026665, 'data_size_processed': 1000, 'model_provider': 'gemini'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 58 | 5 | concurrent_test | llama-3.1-8b | 1144.2910194396973 | 0.8738903631276332 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346161.882389 | 0.0010240000000000002 | 5120 | 0.7805684315735588 | {'min_latency_ms': 594.846248626709, 'max_latency_ms': 1759.0994834899902, 'p95_latency_ms': np.float64(1631.7564606666563), 'p99_latency_ms': np.float64(1733.6308789253235), 'total_time_s': 11.443083047866821, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010240000000000002, 'quality_std': 0.0613021253594286, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 59 | 5 | concurrent_test | llama-3.1-8b | 1128.666615486145 | 3.527006383973853 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346164.7190907 | 0.0010240000000000002 | 5120 | 0.7915276538063776 | {'min_latency_ms': 610.3026866912842, 'max_latency_ms': 1934.2899322509766, 'p95_latency_ms': np.float64(1909.2738270759583), 'p99_latency_ms': np.float64(1929.286711215973), 'total_time_s': 2.835265636444092, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010240000000000002, 'quality_std': 0.055242108041169316, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 60 | 5 | concurrent_test | llama-3.1-70b | 1341.410732269287 | 0.7454805363345477 | 0.0 | 0.0 | 1.0 | 0 | 10 | 1 | 1759346178.2571824 | 0.004096000000000001 | 5120 | 0.8513858389112968 | {'min_latency_ms': 566.3845539093018, 'max_latency_ms': 1769.1750526428223, 'p95_latency_ms': np.float64(1743.9924359321594), 'p99_latency_ms': np.float64(1764.1385293006897), 'total_time_s': 13.414166450500488, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0004096000000000001, 'quality_std': 0.06286695897481548, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 61 | 5 | concurrent_test | llama-3.1-70b | 1410.3811264038086 | 3.52022788340447 | 0.0 | 0.0 | 1.0 | 0 | 10 | 6 | 1759346181.0992308 | 0.004096000000000001 | 5120 | 0.8534058400920448 | {'min_latency_ms': 572.9773044586182, 'max_latency_ms': 1928.0850887298584, 'p95_latency_ms': np.float64(1903.529143333435), 'p99_latency_ms': np.float64(1923.1738996505737), 'total_time_s': 2.8407251834869385, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0004096000000000001, 'quality_std': 0.059750620144052545, 'data_size_processed': 1000, 'model_provider': 'llama'} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 62 | 5 | memory_test | gpt-4o-mini | 1177.2440481185913 | 3.97501008701798 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346193.7901201 | 0.0038400000000000023 | 25600 | 0.8512259391579574 | {'min_latency_ms': 537.5485420227051, 'max_latency_ms': 2001.0862350463867, 'p95_latency_ms': np.float64(1892.5400853157041), 'p99_latency_ms': np.float64(1985.4257130622864), 'total_time_s': 12.578584432601929, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.680000000000005e-05, 'quality_std': 0.0581968026848211, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 63 | 5 | memory_test | gpt-4o-mini | 1229.8026752471924 | 3.9282369679460363 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346206.6300905 | 0.0038400000000000023 | 25600 | 0.8537868196468017 | {'min_latency_ms': 518.6026096343994, 'max_latency_ms': 1944.331407546997, 'p95_latency_ms': np.float64(1909.6850633621214), 'p99_latency_ms': np.float64(1940.652117729187), 'total_time_s': 12.72835636138916, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.680000000000005e-05, 'quality_std': 0.05181407518487485, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 64 | 5 | memory_test | gpt-4o-mini | 1274.8144483566284 | 3.7483119966709824 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346220.0900073 | 0.0038400000000000023 | 25600 | 0.8487480924622282 | {'min_latency_ms': 529.292106628418, 'max_latency_ms': 1996.4158535003662, 'p95_latency_ms': np.float64(1960.6919050216675), 'p99_latency_ms': np.float64(1988.2149648666382), 'total_time_s': 13.339337825775146, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 7.680000000000005e-05, 'quality_std': 0.05812899461310237, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 65 | 5 | memory_test | gpt-4o | 1174.5057010650635 | 4.0514136389986115 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346232.557784 | 0.12800000000000017 | 25600 | 0.9484191580718665 | {'min_latency_ms': 286.58127784729004, 'max_latency_ms': 1877.345085144043, 'p95_latency_ms': np.float64(1735.1435780525208), 'p99_latency_ms': np.float64(1842.000467777252), 'total_time_s': 12.341371297836304, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.8046875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0025600000000000032, 'quality_std': 0.0491398572941036, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 66 | 5 | memory_test | gpt-4o | 1225.388593673706 | 3.875932429633176 | 0.125 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346245.5669534 | 0.12800000000000017 | 25600 | 0.9557179217710832 | {'min_latency_ms': 514.6803855895996, 'max_latency_ms': 2034.6620082855225, 'p95_latency_ms': np.float64(1909.4360709190366), 'p99_latency_ms': np.float64(2010.34743309021), 'total_time_s': 12.900121688842773, 'initial_memory_mb': 294.8046875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0025600000000000032, 'quality_std': 0.04870463047338363, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 67 | 5 | memory_test | gpt-4o | 1244.0021991729736 | 3.7266446101546777 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346259.1414776 | 0.12800000000000017 | 25600 | 0.9458944372937584 | {'min_latency_ms': 521.9912528991699, 'max_latency_ms': 1986.6855144500732, 'p95_latency_ms': np.float64(1953.3554077148438), 'p99_latency_ms': np.float64(1978.9683985710144), 'total_time_s': 13.416895151138306, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0025600000000000032, 'quality_std': 0.04851286804634898, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 68 | 5 | memory_test | gpt-4-turbo | 1181.3615322113037 | 4.124998416603219 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346271.374578 | 0.25600000000000034 | 25600 | 0.9651345363111258 | {'min_latency_ms': 353.2071113586426, 'max_latency_ms': 1966.524362564087, 'p95_latency_ms': np.float64(1945.0057744979858), 'p99_latency_ms': np.float64(1965.7717752456665), 'total_time_s': 12.121216773986816, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0051200000000000065, 'quality_std': 0.04338778763022959, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 69 | 5 | memory_test | gpt-4-turbo | 1291.4055681228638 | 3.77552400952112 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346284.731812 | 0.25600000000000034 | 25600 | 0.9689389907566063 | {'min_latency_ms': 555.095911026001, 'max_latency_ms': 2027.0910263061523, 'p95_latency_ms': np.float64(1966.5393114089964), 'p99_latency_ms': np.float64(2018.9284563064575), 'total_time_s': 13.243194818496704, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0051200000000000065, 'quality_std': 0.04154143035607859, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 70 | 5 | memory_test | gpt-4-turbo | 1261.4208269119263 | 3.663208321130074 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346298.4905493 | 0.25600000000000034 | 25600 | 0.9573488473081913 | {'min_latency_ms': 284.8320007324219, 'max_latency_ms': 2011.866807937622, 'p95_latency_ms': np.float64(1975.5298137664795), 'p99_latency_ms': np.float64(2000.7115292549133), 'total_time_s': 13.649237394332886, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0051200000000000065, 'quality_std': 0.04380501534660363, 'data_size_processed': 1000, 'model_provider': 'gpt', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 71 | 5 | memory_test | claude-3-5-sonnet | 1270.3543138504028 | 3.7944320989090614 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346311.7936022 | 0.07680000000000001 | 25600 | 0.948463600922609 | {'min_latency_ms': 622.9770183563232, 'max_latency_ms': 1970.0510501861572, 'p95_latency_ms': np.float64(1868.455410003662), 'p99_latency_ms': np.float64(1957.5506472587585), 'total_time_s': 13.177202463150024, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.001536, 'quality_std': 0.04872900892927657, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 72 | 5 | memory_test | claude-3-5-sonnet | 1154.527621269226 | 4.107802148818313 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346324.0782034 | 0.07680000000000001 | 25600 | 0.9535056752128789 | {'min_latency_ms': 526.8404483795166, 'max_latency_ms': 1841.3877487182617, 'p95_latency_ms': np.float64(1815.3946280479431), 'p99_latency_ms': np.float64(1837.1384692192078), 'total_time_s': 12.171959161758423, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.001536, 'quality_std': 0.04600056992617095, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 73 | 5 | memory_test | claude-3-5-sonnet | 1341.6658163070679 | 3.5050325493977805 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346338.4560573 | 0.07680000000000001 | 25600 | 0.947231761746643 | {'min_latency_ms': 607.1841716766357, 'max_latency_ms': 1968.3496952056885, 'p95_latency_ms': np.float64(1938.420307636261), 'p99_latency_ms': np.float64(1963.8122081756592), 'total_time_s': 14.265202760696411, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 294.9296875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.001536, 'quality_std': 0.0468041040494112, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 74 | 5 | memory_test | claude-3-haiku | 1268.9041805267334 | 3.6527405734902607 | 0.125 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346352.2760284 | 0.06400000000000008 | 25600 | 0.8657832919908838 | {'min_latency_ms': 576.9007205963135, 'max_latency_ms': 1978.3263206481934, 'p95_latency_ms': np.float64(1900.9657382965088), 'p99_latency_ms': np.float64(1977.4397349357605), 'total_time_s': 13.688352346420288, 'initial_memory_mb': 294.9296875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0012800000000000016, 'quality_std': 0.05791027367020173, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 75 | 5 | memory_test | claude-3-haiku | 1273.6989831924438 | 3.7602543777430877 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346365.681829 | 0.06400000000000008 | 25600 | 0.8396294693060197 | {'min_latency_ms': 521.7316150665283, 'max_latency_ms': 1988.7199401855469, 'p95_latency_ms': np.float64(1945.9344744682312), 'p99_latency_ms': np.float64(1987.1683859825134), 'total_time_s': 13.296972751617432, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0012800000000000016, 'quality_std': 0.06291349263235946, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 76 | 5 | memory_test | claude-3-haiku | 1234.9269914627075 | 3.9335082345318124 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346378.5192664 | 0.06400000000000008 | 25600 | 0.8469784358915146 | {'min_latency_ms': 529.503345489502, 'max_latency_ms': 1981.7008972167969, 'p95_latency_ms': np.float64(1859.1547846794128), 'p99_latency_ms': np.float64(1963.3227896690369), 'total_time_s': 12.711299180984497, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0012800000000000016, 'quality_std': 0.061722943046806616, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 77 | 5 | memory_test | claude-3-sonnet | 1195.9008169174194 | 4.06962738382444 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346390.9144897 | 0.3840000000000003 | 25600 | 0.9026531444228556 | {'min_latency_ms': -36.6673469543457, 'max_latency_ms': 1991.610050201416, 'p95_latency_ms': np.float64(1819.4202184677124), 'p99_latency_ms': np.float64(1987.222683429718), 'total_time_s': 12.286137104034424, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000005, 'quality_std': 0.058229589360407986, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 78 | 5 | memory_test | claude-3-sonnet | 1372.0379829406738 | 3.502253345465805 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346405.3043494 | 0.3840000000000003 | 25600 | 0.8837364473272626 | {'min_latency_ms': 543.1270599365234, 'max_latency_ms': 1992.779016494751, 'p95_latency_ms': np.float64(1931.822681427002), 'p99_latency_ms': np.float64(1987.4089169502258), 'total_time_s': 14.276522874832153, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000005, 'quality_std': 0.05634614113838598, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 79 | 5 | memory_test | claude-3-sonnet | 1257.2709035873413 | 3.7764857062182706 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346418.6521854 | 0.3840000000000003 | 25600 | 0.9053414058751514 | {'min_latency_ms': 529.8404693603516, 'max_latency_ms': 1990.1280403137207, 'p95_latency_ms': np.float64(1911.1806631088257), 'p99_latency_ms': np.float64(1976.6331052780151), 'total_time_s': 13.239822387695312, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.007680000000000005, 'quality_std': 0.050506656009957705, 'data_size_processed': 1000, 'model_provider': 'claude', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 80 | 5 | memory_test | gemini-1.5-pro | 1221.5951490402222 | 3.8372908969845323 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346431.7921565 | 0.03200000000000004 | 25600 | 0.9365925291921394 | {'min_latency_ms': 329.1811943054199, 'max_latency_ms': 1995.384693145752, 'p95_latency_ms': np.float64(1965.0332808494568), 'p99_latency_ms': np.float64(1988.3063769340515), 'total_time_s': 13.030025959014893, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0006400000000000008, 'quality_std': 0.04847128641002876, 'data_size_processed': 1000, 'model_provider': 'gemini', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 81 | 5 | memory_test | gemini-1.5-pro | 1351.8355464935303 | 3.6227975436552606 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346445.7126448 | 0.03200000000000004 | 25600 | 0.9323552590826123 | {'min_latency_ms': 515.129566192627, 'max_latency_ms': 2008.0702304840088, 'p95_latency_ms': np.float64(1958.6564779281616), 'p99_latency_ms': np.float64(2004.1296029090881), 'total_time_s': 13.801488876342773, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0006400000000000008, 'quality_std': 0.055840796126395656, 'data_size_processed': 1000, 'model_provider': 'gemini', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 82 | 5 | memory_test | gemini-1.5-pro | 1240.622534751892 | 3.8813384098374453 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346458.7192729 | 0.03200000000000004 | 25600 | 0.9407390543744837 | {'min_latency_ms': -29.146671295166016, 'max_latency_ms': 1934.4398975372314, 'p95_latency_ms': np.float64(1849.7230291366577), 'p99_latency_ms': np.float64(1918.0084466934204), 'total_time_s': 12.8821542263031, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0006400000000000008, 'quality_std': 0.050597003908357786, 'data_size_processed': 1000, 'model_provider': 'gemini', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 83 | 5 | memory_test | gemini-1.5-flash | 1237.6702642440796 | 3.812923495644346 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346471.9588974 | 0.019200000000000002 | 25600 | 0.8556073429019542 | {'min_latency_ms': 536.4787578582764, 'max_latency_ms': 2010.1728439331055, 'p95_latency_ms': np.float64(1911.8669629096985), 'p99_latency_ms': np.float64(1976.080708503723), 'total_time_s': 13.113297462463379, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.000384, 'quality_std': 0.06082135675952047, 'data_size_processed': 1000, 'model_provider': 'gemini', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 84 | 5 | memory_test | gemini-1.5-flash | 1180.0980806350708 | 4.016049090832003 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346484.5327744 | 0.019200000000000002 | 25600 | 0.8718428063415768 | {'min_latency_ms': 109.58051681518555, 'max_latency_ms': 1993.358850479126, 'p95_latency_ms': np.float64(1872.3165988922117), 'p99_latency_ms': np.float64(1992.416422367096), 'total_time_s': 12.450047016143799, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.000384, 'quality_std': 0.0613916834940056, 'data_size_processed': 1000, 'model_provider': 'gemini', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 85 | 5 | memory_test | gemini-1.5-flash | 1194.4490098953247 | 4.009936119483076 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346497.1201088 | 0.019200000000000002 | 25600 | 0.8652112059805899 | {'min_latency_ms': 520.3211307525635, 'max_latency_ms': 1942.4259662628174, 'p95_latency_ms': np.float64(1834.6370577812195), 'p99_latency_ms': np.float64(1890.3984904289243), 'total_time_s': 12.469026565551758, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.000384, 'quality_std': 0.05312368368226588, 'data_size_processed': 1000, 'model_provider': 'gemini', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 86 | 5 | memory_test | llama-3.1-8b | 1306.2016773223877 | 3.683763547696555 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346510.812732 | 0.005119999999999998 | 25600 | 0.7727309350554936 | {'min_latency_ms': 527.4953842163086, 'max_latency_ms': 1997.086524963379, 'p95_latency_ms': np.float64(1942.7793741226194), 'p99_latency_ms': np.float64(1994.0643763542175), 'total_time_s': 13.573075294494629, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010239999999999995, 'quality_std': 0.05596283861854901, 'data_size_processed': 1000, 'model_provider': 'llama', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 87 | 5 | memory_test | llama-3.1-8b | 1304.1251468658447 | 3.617383744773005 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346524.7711937 | 0.005119999999999998 | 25600 | 0.785787220179362 | {'min_latency_ms': 112.00571060180664, 'max_latency_ms': 2015.146255493164, 'p95_latency_ms': np.float64(2001.4938592910767), 'p99_latency_ms': np.float64(2012.321424484253), 'total_time_s': 13.822144269943237, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010239999999999995, 'quality_std': 0.0552285639827787, 'data_size_processed': 1000, 'model_provider': 'llama', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 88 | 5 | memory_test | llama-3.1-8b | 1290.5346298217773 | 3.671522710311051 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346538.5084107 | 0.005119999999999998 | 25600 | 0.7771978709125356 | {'min_latency_ms': 565.7510757446289, 'max_latency_ms': 1945.1093673706055, 'p95_latency_ms': np.float64(1906.785237789154), 'p99_latency_ms': np.float64(1942.4526476860046), 'total_time_s': 13.618327856063843, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.00010239999999999995, 'quality_std': 0.057252814774054535, 'data_size_processed': 1000, 'model_provider': 'llama', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 89 | 5 | memory_test | llama-3.1-70b | 1213.9334726333618 | 3.947675276737486 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346551.2951744 | 0.02047999999999999 | 25600 | 0.8683286341213061 | {'min_latency_ms': -79.86569404602051, 'max_latency_ms': 2014.9149894714355, 'p95_latency_ms': np.float64(1919.9433565139768), 'p99_latency_ms': np.float64(1992.4925136566162), 'total_time_s': 12.665682077407837, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0004095999999999998, 'quality_std': 0.05862810413022958, 'data_size_processed': 1000, 'model_provider': 'llama', 'iteration': 0} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 90 | 5 | memory_test | llama-3.1-70b | 1298.1958770751953 | 3.7049711897976763 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346564.9280033 | 0.02047999999999999 | 25600 | 0.8889975698232048 | {'min_latency_ms': 503.5574436187744, 'max_latency_ms': 2020.4124450683594, 'p95_latency_ms': np.float64(1901.4497756958008), 'p99_latency_ms': np.float64(1986.3133001327512), 'total_time_s': 13.495381593704224, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0004095999999999998, 'quality_std': 0.053463278827038344, 'data_size_processed': 1000, 'model_provider': 'llama', 'iteration': 1} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |
| 91 | 5 | memory_test | llama-3.1-70b | 1187.040138244629 | 4.165139112812611 | 0.0 | 0.0 | 1.0 | 0 | 50 | 5 | 1759346577.0467978 | 0.02047999999999999 | 25600 | 0.8884529182459214 | {'min_latency_ms': 506.2377452850342, 'max_latency_ms': 2026.6106128692627, 'p95_latency_ms': np.float64(1958.3556652069092), 'p99_latency_ms': np.float64(2007.5032830238342), 'total_time_s': 12.004400968551636, 'initial_memory_mb': 295.0546875, 'final_memory_mb': 295.0546875, 'avg_tokens_per_request': 512.0, 'cost_per_request': 0.0004095999999999998, 'quality_std': 0.05625669416735748, 'data_size_processed': 1000, 'model_provider': 'llama', 'iteration': 2} | 0.0 | 0.0 | 0.0 | 0.0 | 0 | False | 0 | 0.0 | 0 | 0.0 | False |