{"id":67,"date":"2020-01-27T20:58:09","date_gmt":"2020-01-27T20:58:09","guid":{"rendered":"http:\/\/groups.cs.umass.edu\/kdl\/?page_id=67"},"modified":"2020-01-27T21:03:49","modified_gmt":"2020-01-27T21:03:49","slug":"causal-eval-data","status":"publish","type":"page","link":"https:\/\/groups.cs.umass.edu\/kdl\/causal-eval-data\/","title":{"rendered":"Evaluating Causal Discovery Algorithms with Software Systems"},"content":{"rendered":"
\n
\n

To obtain realistic data for evaluating causal discovery algorithms, we designed and executed factorial experiments on three large-scale software systems: PostgreSQL, the Java Development Kit, and web server infrastructure. Each domain is characterized by three classes of variables: covariates, treatments, and outcomes. Under the factorial experiment design, outcomes were measured for every combination of subjects and treatments. This yields a dataset with many records for the same subject. To permit a variety of data transformation opportunities, we performed multiple trials of each factorial experiment.<\/p>\n

\n
\n

PostgreSQL<\/h3>\n

We collected a sample of StackOverflow<\/a>‘s database, and gathered a sample of user-authored queries from their query explorer<\/a>. We executed each query against our sample of the database, varying database configuration and monitoring query execution.<\/p>\n

11,252 subjects
\n3 treatments
\n7 outcomes
\n10 covariates<\/p>\n<\/div>\n

\n

JDK<\/h3>\n

We downloaded a sample of Maven<\/a>-enabled Java projects from GitHub<\/a>. We compiled and ran the unit tests of each project, varying JDK options and monitoring runtime behavior.<\/p>\n

473 subjects
\n3 treatments
\n5 outcomes
\n5 covariates<\/p>\n<\/div>\n

\n

Networking<\/h3>\n

We identified a pool of websites using a small web crawl. We then executed several web requests against each site, varying request parameters and monitoring output.<\/p>\n

2,599 subjects
\n3 treatments
\n5 outcomes
\n1 covariate<\/p>\n<\/div>\n<\/div>\n

\n
\n Download (212M)<\/a>
\nDetails
\n<\/span>
\n<\/a><\/div>\n
\n Download (3.2M)<\/a>
\nDetails
\n<\/span>
\n<\/a><\/div>\n
\n Download (78M)<\/a>
\nDetails
\n<\/span>
\n<\/a><\/div>\n<\/div>\n
\n
\n

PostgreSQL Details<\/h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Field Name<\/th>\nCategory<\/th>\nDescription<\/th>\n<\/tr>\n<\/thead>\n
url<\/td>\nsubject identifier<\/td>\nIdentifies the source of the query<\/td>\n<\/tr>\n
trial<\/td>\ntrial identifier<\/td>\n<\/td>\n<\/tr>\n
index_level<\/td>\ntreatment<\/td>\nIndicates the level of indexing employed.
\n0: No indexing
\n1: Indexing on primary keys and foreign keys only
\n2: Indexing on primary keys, foreign keys, and other commonly-referenced fields<\/td>\n<\/tr>\n
page_cost<\/td>\ntreatment<\/td>\nIndicates the estimated disk access cost provided to the Postgres query planner. 0 corresponds the smallest disk access cost, 3 corresponds the largest disk access cost.<\/td>\n<\/tr>\n
memory_level<\/td>\ntreatment<\/td>\nThe amount of working memory provided to Postgres, in increasing increments from 0 to 2<\/td>\n<\/tr>\n
local_written_blocks<\/td>\noutcome<\/td>\nThe number of blocks written to temporary tables andindices<\/td>\n<\/tr>\n
temp_written_blocks<\/td>\noutcome<\/td>\nThe number of blocks written to short-term working memory<\/td>\n<\/tr>\n
shared_hit_blocks<\/td>\noutcome<\/td>\nNumber of regular table blocks hit in the cache<\/td>\n<\/tr>\n
temp_read_blocks<\/td>\noutcome<\/td>\nNumber of blocks read from temporary tables and indices<\/td>\n<\/tr>\n
local_read_blocks<\/td>\noutcome<\/td>\nNumber of blocks read from temporary tables and indices<\/td>\n<\/tr>\n
runtime<\/td>\noutcome<\/td>\nRuntime of the query in milliseconds<\/td>\n<\/tr>\n
shared_read_blocks<\/td>\noutcome<\/td>\nNumber of blocks read from regular tables and indices<\/td>\n<\/tr>\n
rows<\/td>\ncovariate<\/td>\nThe number of rows produced by the query<\/td>\n<\/tr>\n
creation_year<\/td>\ncovariate<\/td>\nYear the query was created on the StackOverflow site.<\/td>\n<\/tr>\n
num_ref_tables<\/td>\ncovariate<\/td>\nNumber of tables referenced by the query<\/td>\n<\/tr>\n
num_joins<\/td>\ncovariate<\/td>\nNumber of joins employed by the query<\/td>\n<\/tr>\n
num_group_by<\/td>\ncovariate<\/td>\nNumber of “group by” clauses employed by the query<\/td>\n<\/tr>\n
queries_by_user<\/td>\ncovariate<\/td>\nThe number of other queries written by the author of this query<\/td>\n<\/tr>\n
length_chars<\/td>\ncovariate<\/td>\nLength of the query in characters<\/td>\n<\/tr>\n
total_ref_rows<\/td>\ncovariate<\/td>\nTotal number of rows of all tables referenced in the query<\/td>\n<\/tr>\n
local_hit_blocks<\/td>\ncovariate<\/td>\nNumber of temporary table and temporary index blocks hit in the cache<\/td>\n<\/tr>\n
favorite_count<\/td>\ncovariate<\/td>\nNumber of times another StackOverflow user has marked this query as a favorite<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
\n
\n

JDK Details<\/h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Field Name<\/th>\nCategory<\/th>\nDescription<\/th>\n<\/tr>\n<\/thead>\n
repo_name<\/td>\nsubject identifier<\/td>\nName of the GitHub repository containing the experimentation code<\/td>\n<\/tr>\n
trial<\/td>\ntrial identifier<\/td>\n<\/td>\n<\/tr>\n
debug<\/td>\ntreatment<\/td>\nIndicates whether debug symbols were requested during compilation<\/td>\n<\/tr>\n
obfuscate<\/td>\ntreatment<\/td>\nIndicates whether a code obfuscator was run on the final JAR file<\/td>\n<\/tr>\n
parallelgc<\/td>\ntreatment<\/td>\nIndicates whether a parallel garbage collection was employed during execution (instead of serial garbage collection)<\/td>\n<\/tr>\n
num_bytecode_ops<\/td>\noutcome<\/td>\nNumber of bytecode instructions in the compiled code<\/td>\n<\/tr>\n
total_unit_test_time<\/td>\noutcome<\/td>\nNumber of seconds required to execute unit tests<\/td>\n<\/tr>\n
allocated_bytes<\/td>\noutcome<\/td>\nNumber of bytes allocated during execution of unit tests<\/td>\n<\/tr>\n
jar_file_size_bytes<\/td>\noutcome<\/td>\nSize of JAR file after compilation (and possibly obfuscation)<\/td>\n<\/tr>\n
compile_time_ms<\/td>\noutcome<\/td>\nNumber of milliseconds to compile the source<\/td>\n<\/tr>\n
source_ncss<\/td>\ncovariate<\/td>\nNumber of non-comment source statements in the source code<\/td>\n<\/tr>\n
test_classes<\/td>\ncovariate<\/td>\nNumber of Java classes in the unit test source<\/td>\n<\/tr>\n
test_functions<\/td>\ncovariate<\/td>\nNumber of functions in the unit test source<\/td>\n<\/tr>\n
test_ncss<\/td>\ncovariate<\/td>\nNumber of non-comment source statements in the test source<\/td>\n<\/tr>\n
test_javadocs<\/td>\ncovariate<\/td>\nNumber of JavaDoc comments in the test source<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
\n
\n

Networking Details<\/h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Field Name<\/th>\nCategory<\/th>\nDescription<\/th>\n<\/tr>\n<\/thead>\n
url<\/td>\nsubject identifier<\/td>\nWeb address requested<\/td>\n<\/tr>\n
trial<\/td>\ntrial identifier<\/td>\n<\/td>\n<\/tr>\n
mobile_user_agent<\/td>\ntreatment<\/td>\nIndicates if the site was requested with a mobile user agent<\/td>\n<\/tr>\n
proxy<\/td>\ntreatment<\/td>\nIndicates if the site was requested through a proxy server<\/td>\n<\/tr>\n
compression<\/td>\ntreatment<\/td>\nIndicates if the request indicates that compression is supported<\/td>\n<\/tr>\n
html_attrs<\/td>\noutcome<\/td>\nNumber of HTML attributes in the response<\/td>\n<\/tr>\n
html_tags<\/td>\noutcome<\/td>\nNumber of HTML tags in the response<\/td>\n<\/tr>\n
elapsed<\/td>\noutcome<\/td>\nElapsed time between request and response in seconds<\/td>\n<\/tr>\n
decompressed_content_length<\/td>\noutcome<\/td>\nLength of the response content, in bytes, after decompression<\/td>\n<\/tr>\n
raw_content_length<\/td>\noutcome<\/td>\nLength of the response content, in bytes, before decompression<\/td>\n<\/tr>\n
server.class<\/td>\ncovariate<\/td>\nCategory of web server issuing the response<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

To obtain realistic data for evaluating causal discovery algorithms, we designed and executed factorial experiments on three large-scale software systems: PostgreSQL, the Java Development Kit, and web server infrastructure. Each domain is characterized by three classes of variables: covariates, treatments, and outcomes. Under the factorial experiment design, outcomes were measured for every combination of subjects … Continue reading “Evaluating Causal Discovery Algorithms with Software Systems”<\/span><\/a><\/p>\n","protected":false},"author":24,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-67","page","type-page","status-publish","hentry","hfeed"],"_links":{"self":[{"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/pages\/67","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/comments?post=67"}],"version-history":[{"count":3,"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/pages\/67\/revisions"}],"predecessor-version":[{"id":71,"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/pages\/67\/revisions\/71"}],"wp:attachment":[{"href":"https:\/\/groups.cs.umass.edu\/kdl\/wp-json\/wp\/v2\/media?parent=67"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}