---
title: "Part 2: Deconstruction Wood Sensitivity Analysis"
subtitle: "Deconstruction Data Preparation using OSU average softwood density factor"
author:
- Andey Nunes, MS
- Research Analyst 2
- Oregon DEQ
date: "`r format(Sys.time(), '%B %d, %Y')`"
output:
  html_document:
    df_print: paged
    toc: yes
    toc_depth: '3'
---

# Deconstruction data preparation

## Introduction and Project Goals

The 2018 Deconstruction Data Analysis uses project data from residential single family homes removed under a City of Portland Deconstruction permit. The goal of the data analysis is to quantify the net environmental benefits resulting from avoided disposal of materials due to salvage/reuse (measured as Global Warming Potential and Primary Energy Demand impacts). 

Figure 1 illustrates the workflow process for the analysis.

![####FIGURE 1: Project Workflow](data/project_workflow.png)

This data preparation notebook uses the `decon_material_weight.csv` output from the `01_decon_material_weight_conversion.Rmd` file and additional input data sources to prepare a data file with impacts associated with the line item of materials reported in the reciepts. Also included are some optional basic data structure exploration routines which may be of interest to analysts and stakeholders, but are not directly involved in the reporting of environmental impacts.  

##### Notebook Setup
The analysis is intended to be reproducible using R coding procedures and employs the following R Packages:
```{r packages, echo=F}
# create a list of packages to be installed and active (required) for use in the notebook and resulting reports
packages <- c( "fBasics", "ggthemes", "grDevices", "knitr", "rebus", "rstudioapi", "scales", "tidyverse")
# print this list as a table
knitr::kable(packages, col.names = "package name")
```
Prior to executing code and producing outputs, the packages must be installed and accessed, and basic features of the document set up by specifying code chunk defaults.
```{r load packages, warning = F, echo=FALSE}
# set defalut code chunks to "echo=TRUE" to display the code chunk inline with the narrative text; and report numbers to a default of three significant digits.
knitr::opts_chunk$set(echo = TRUE)
options(digits = 1)
options(scipen = 999)

lapply(packages, require, character.only = T)
options(xtable.comment = F)

# DEQ color palette for graphics using approximate match selection from grDevices package

DEQ_pal <- c('aquamarine4', 'steelblue4', 'lightseagreen', 'yellowgreen', 'darkorange1',
            'darkseagreen3', 'slateblue1', 'powderblue', 'khaki', 'lightsalmon',
            'seagreen4', 'deepskyblue4', 'darkslateblue', 'magenta4', 'palegreen4',
            'cyan4', 'goldenrod2', 'indianred3', 'seagreen')


# clean up workspace
rm(packages)

```

### Notebook Contents and Output

This notebook includes some optional basic data structure exploration, which is indicated using the `include =`  argument inside the code chunk title string (curly braces after the code chunk opening ticks) Users can deselect any optional output in this notebook by changing the  `include =` argument from `TRUE` to `FALSE`. Additional summary information may also be selected to print by removing the comment `#` character from in front of the summary call in the appropriate code chunks.

Data file output with descriptions are given in the following table. All output resides inside the R project folder:

  
####TABLE 1: Data Preparation File Output Objects
  
| Output Name |                                 Description                                                    |
|-------------|------------------------------------------------------------------------------------------------|
| `dropbox_EOL_weight_composition.csv` | EOL weight (kg) for assigned composition of dropbox/dropbox materials |
| `decon_house_weights.csv` | project based material weights converted to 'kg' for impact calculations         |
|`deconMaterialName_SimpleEOLname_mapping.csv` | mapping of different material naming schemes used for data wrangling |
|`deconData.csv` | deconstruction scenario impacts by project, material, and EOL disposition including dropbox |
|`demoData.csv` | demolition scenario impacts by project, material, and EOL disposition including dropbox      |



# Data Import

```{r load data}
# from 01_decon_material_weight_conversion.Rmd
decon_material_weight <- read_csv("intermediary/decon_material_weight.csv")
# remove index and reformat project and contractor factors to character vectors
decon_material_weight <- decon_material_weight[,-1]
decon_material_weight$project <- as.character(decon_material_weight$project)


# sourced from DEQ contact Palmeri.Jordan@deq.state.or.us
impact_data <- read_csv("data/LCA impact data for Decon Tool.csv")
EOL <- read_csv("data/EOL.csv")
# for sensitivity analyses, replace this line with the appropriate alternative EOL file
# EOL <- read_csv("data/EOLalt.csv")
```

## Projects/Houses

```{r project characteristics,  warning=FALSE}
# create a new project characteristics data table, coerce project and contractor column types to character factors, and optionally print a summary
project_characteristics <- distinct(select(decon_material_weight, c("project", "house_age", "house_size")))
# summary(project_characteristics)

# calculate and save mean average of the house age and size statistics
average_house_age <- round(mean(project_characteristics$house_age))
average_house_size <- round(mean(project_characteristics$house_size))
number_of_houses <- round(length(project_characteristics$project))
```

There are `r number_of_houses` projects in the data set.  The average house was `r average_house_age` years old and roughly `r average_house_size` square feet. 

```{r reformat data}
# split the full set of decon_material_weight data into two data frames: anything that didn't go into a dropbox vs anything that did! 
salvage_materials_weight <- decon_material_weight %>% 
   filter(deconMaterialName != "dropbox") %>%
   mutate(US_lbs = converted_quantity * 2.2046)

dropbox_weight <- decon_material_weight %>% 
   filter(deconMaterialName == "dropbox") %>%
   select(c("project", "contractor", "house_age", "house_size", "deconMaterialName", "converted_quantity", "converted_quantity_units") ) %>%
   group_by(contractor, project) %>%
   summarise(total_dropbox_quantity = sum(converted_quantity)) %>%
   add_column(quantity_units = "kg")

rm(decon_material_weight)
```

The data is sum totaled by project to get a total weight of all salvaged materials and total weight of all dropbox tickets per project. These salvage and dropbox totaled weight values are added to get a combined `total_house_weight` which serves as an estimate the weight of the house without the foundation. This in turn is used to generate a percentage of house material salvaged by weight for each project. 

```{r house weight}
# save project weight objects
reuse_by_project <- salvage_materials_weight %>%
   group_by(contractor, project) %>%
   summarise(total_salvage_quantity = sum(converted_quantity)) %>%
   add_column(quantity_units = "kg")

house_weight <- full_join(reuse_by_project, dropbox_weight, by = c("project", "contractor")) %>%
   mutate(total_house_weight = total_salvage_quantity + total_dropbox_quantity) %>%
   mutate(total_house_weight_units = "kg") %>%
   mutate(percent_salvaged = (total_salvage_quantity / total_house_weight * 100)) %>%
   rename(salvage_quantity_units = quantity_units.x) %>%
   rename(dropbox_quantity_units = quantity_units.y) %>%
   left_join(project_characteristics, by = "project") %>%
   arrange(desc(percent_salvaged)) %>%
   group_by(contractor)

# save output for later use 

write_csv(house_weight, "intermediary/house_weight.csv")

# clear the environment of unneeded objects
rm(reuse_by_project)

summary(select(house_weight, c("total_salvage_quantity", "total_dropbox_quantity", "total_house_weight", "percent_salvaged", "house_age", "house_size") ))
```


The average house yielded `r round(mean(house_weight$total_salvage_quantity))*2.20462` pounds of materials for salvage and `r round(mean(house_weight$total_dropbox_quantity))*2.20462` pounds of materials in dropboxes, which results in the (arithmetic mean) average of `r round(mean(house_weight$percent_salvaged))` percent material salvage by weight for this set of deconstruction projects.

### Data visualization
A few initial data visualizations are explored in the following code chunks, most of which if included in a final report, will end up in an appendix.
```{r graphics,  include = F}
# graphics are provided here for exploratory purposes, these are also reproduced as appendix diagrams in the final report with weight converted from kg to US lbs

ggplot(house_weight, aes(contractor, percent_salvaged)) +
   geom_jitter(size = 2, alpha = 0.4,  width = 0.1) +
   labs(x = "contractor", y = "percentage by weight salvaged") +
   theme_minimal() +
   ggtitle("Percentage materials salvaged by contractor") 

ggplot(house_weight, aes(house_size, total_salvage_quantity)) +
   geom_point( aes(color = contractor), size = 2,  position = "jitter") +
   geom_smooth(method = "glm") +
   scale_color_calc() +
   labs(x = "house size in square feet", y = "kg salvaged materials") +
   theme_minimal() +
   ggtitle("Materials salvaged by house size")

ggplot(house_weight, aes(house_size, total_dropbox_quantity)) +
   geom_point(aes(color = contractor), size = 2,  position = "jitter") +
   geom_smooth(method = "glm") +
   scale_color_calc() +
   labs(x = "house size in square feet", y = "kg dropbox materials") +
   theme_minimal() +
   ggtitle("Disposal weight by house size")

ggplot(house_weight, aes(house_size, percent_salvaged, color = contractor)) +
   geom_point(size = 2) +
   geom_smooth(method = "glm") +
   scale_color_calc() +
   labs(x = "house size in square feet", y = "percentage by weight salvaged") +
   theme_minimal() +
   ggtitle("Percentage materials salvaged by house size")

ggplot(house_weight, aes(house_age, percent_salvaged, color = contractor)) +
   geom_point(size = 2) +
   geom_smooth(method = "glm") +
   scale_color_calc() +
   labs(x = "house age in years old", y = "percentage by weight salvaged") +
   theme_minimal() +
   ggtitle("Percentage materials salvaged by house age")


```

The project team was interested in the relationship between the weight of salvaged and disposal materials and the size (square feet) of the house. The "Materials salvaged by house size" chart shows the scatterplot of these variables, with contractors indicated in the point colors. A positive relationship is expected here, and the plot confirms this by showing sparse instances in the upper left and lower right quadrants of the graph.



```{r project reuse by weight, include = F}
ggplot(salvage_materials_weight, aes(deconMaterialName, project)) +
   geom_tile(aes(fill = US_lbs) ) +
   #scale_fill_distiller(palette = "YlGnBu") +
   scale_fill_gradientn(colours = colorRamps::ygobb(12), name = "weight (lbs)") +
   coord_flip() +
   theme_minimal() +
   labs(x = "", y = "project") +
   theme(axis.text.x = element_blank()) +
   ggtitle("Deconstruction salvaged material quantity by project")

   ggsave("graphs/salvage_weight_heatmap.png", device = "png", width = 7, height = 4)
```


# Impact Data

```{r clean up impacts}

#impact_data <- impact_data %>%
#   select(c(1, 3:7))

# split impacts by disposition to create EOL scenario impact values
reuse_impacts <- impact_data %>%
   filter(.$disposition == "reuse")

demo_impacts <- impact_data %>%
   filter(.$disposition != "reuse")
```

# Data Transformations for Analytical Use 

## Reuse & deconstruction salvage

Impacts on salvaged materials are calculated by joining the `reuse_impacts` to the `salvage_materials_weight` data and taking the product of the `calculated_quantity` and the `impactValue`, which is named `material_impacts` in the `reuse_scenario` data frame.

```{r reuse scenario data}
reuse_scenario <- salvage_materials_weight %>%
   inner_join(reuse_impacts, by = "deconMaterialName") %>%
   mutate(material_impacts = converted_quantity * impactValue) %>%
   mutate(material_impact_units = impactUnit) %>%
   select(c("project", "contractor", "house_age", "house_size", "deconMaterialName", "converted_quantity", "converted_quantity_units", "disposition", "impactCategory", "material_impacts", "material_impact_units")) %>%
   group_by(project, contractor, deconMaterialName, disposition, impactCategory)
 
    
reuse_scenario_impact_summary <- reuse_scenario %>%
   summarise(impact_sum = sum(material_impacts)) %>%
   left_join(project_characteristics, by = "project")  
   
```

Optional preliminary data visualizations of the salvaged materials are included.
```{r reuse scenario data visualizations, include = F}

ggplot(reuse_scenario, aes(x = converted_quantity_units, y = converted_quantity, fill = deconMaterialName) ) +
   geom_bar(stat = "identity", position = "fill") +
   labs(x = "salvage material weight", y = "percentage of weight total") +
   ggtitle("Material percentage of reuse weight")

ggplot(reuse_scenario_impact_summary, aes(x = impactCategory, y = impact_sum, fill = deconMaterialName) ) +
   geom_bar(stat = "identity", position = "fill") +
   labs(x = "salvage material impacts", y = "percentage of impact total") +
   ggtitle("Salvage impacts by material")


# the softwood lumber category overwhelms the rest of the data, so rerun the ggplot calls above with the following data set to see detail of other material categories

reuse_sans_softwood_lumber <- filter(reuse_scenario_impact_summary, deconMaterialName != "softwood lumber")

ggplot(reuse_sans_softwood_lumber, aes(x = impactCategory, y = impact_sum, fill = deconMaterialName) ) +
   geom_bar(stat = "identity", position = "fill") +
   labs(x = "salvage material impacts", y = "percentage of impact total") +
   ggtitle("Salvage impacts by material")

ggplot(filter(reuse_sans_softwood_lumber, impactCategory == "Primary Energy Demand"),  aes(deconMaterialName, impact_sum,  fill = contractor)) +
   geom_bar(stat = "identity") +
   ggtitle("Primary Energy Demand reuse impacts") +
   labs(x = "material", y = "MJ") +
   coord_flip()

rm(reuse_sans_softwood_lumber)
```

## Non-salvage (dropbox/dropbox) material impacts

In order to assign impacts to the dropbox materials, the following material distribution is applied to the total on each project.

```{r dropbox EOL data}
# new deconMaterialName categories to incorporate later, leave commented out until update is released
# new_materials_demo_percentEOL <- EOL[76:86, c(3:4,7:9)]

# dropbox composition percentages to apply to dropbox total weight
dropbox_stream_distribution <- EOL[30:33, 3:5]
dropbox_stream_distribution$Distribution <- as.numeric(str_sub(dropbox_stream_distribution$Distribution, 1, -2))/100

# reshape table for extracting kg by EOL disposition
dropbox_EOL <- EOL[30:33, c(3:4,7:9)] %>%
   gather("Percent Recycled", "Percent Incinerated", "Percent Landfilled",  key = "disposition_assignment", value = "percentage") %>%
   add_column(disposition = case_when(
      .$disposition_assignment == "Percent Recycled" ~ "recyclingGeneric",
      .$disposition_assignment == "Percent Incinerated" ~ "incineration",
      .$disposition_assignment == "Percent Landfilled" ~ "landfill")
      )
# merge the stream composition table with the EOL disposition assignment
dropbox_stream_EOL <- inner_join(dropbox_stream_distribution, dropbox_EOL, by = c("SimpleEOLname", "deconMaterialName")) %>%
   filter(percentage > 0) %>%
   add_column(target_units = rep("kg", length(.$percentage)))

# clean up intermediate objects
rm(dropbox_EOL)
rm(dropbox_stream_distribution)
```

Once these material composition and end-of-life disposition assignments are imposed on the total project dropbox weight, the values essentially become theoretical values subject to changing assumptions or updates in the metro C&D waste composition data.  Each project is assigned the `r length(dropbox_stream_EOL$percentage)` `dropbox_stream_EOL` EOL percentages and then weights calculated for each EOL disposition assignment. Finally the impacts associated with this dropbox EOL assignment profile are calculated and saved as a separate output file. 

```{r dropbox data to dropbox composition & EOL assignment}
dropbox_composition <- full_join(dropbox_weight, dropbox_stream_EOL,  by = c("quantity_units" = "target_units")) %>%
   mutate(kg_composition = total_dropbox_quantity * Distribution) %>%
   mutate(dropbox_EOL_kg = kg_composition * percentage)

write.csv(dropbox_composition, "intermediary/dropbox_EOL_weight_composition.csv")
rm(dropbox_weight)

# assign impacts to the EOL disposition kg and clean up for merge with `reuse_scenario`
dropbox_impacts <- left_join(dropbox_composition, impact_data, by = c("deconMaterialName", "disposition")) %>%
   full_join(project_characteristics) %>%
   mutate(material_impacts = dropbox_EOL_kg * impactValue) %>%
   rename(material_impact_units = impactUnit) %>%
   rename(quantity = dropbox_EOL_kg) %>%
   select("house_age", "house_size", "project", "contractor", "deconMaterialName", "quantity", "quantity_units", "disposition", "impactCategory", "material_impacts", "material_impact_units", "SimpleEOLname")
   

ggplot(dropbox_impacts, aes(x = impactCategory, y = material_impacts, fill = SimpleEOLname) ) +
   geom_bar(stat = "identity", position = "dodge") +
   labs(x = "dropbox material impacts", y = "impact total: MJ for Energy and kg CO2e for GWP", fill = "") +
   scale_y_continuous(labels = comma) +
   scale_fill_manual(values = DEQ_pal[8:11]) +
   theme_tufte() +
   theme(legend.position = "top", legend.direction = "horizontal") +
   ggtitle("Dropbox impacts by material")

```

Before developing the final deconstruction scenario data table, we'll need to bring the `material_category_mapping` into the `reuse_scenario` data frame and combine the `reuse_scenario` data frame with the `dropbox_impacts`. This yields the `decon` data frame of material quantities and impacts by project and material type for all materials presented in the receipts summary.

```{r material mapping}
material_category_mapping <- EOL[1:28,3:4]
write.csv(material_category_mapping, "intermediary/deconMaterialName_SimpleEOLname_mapping.csv")
```


```{r decon data,  warning = F}
decon <- left_join(reuse_scenario, material_category_mapping, by = "deconMaterialName") %>%
   rename(quantity = converted_quantity) %>%
   rename(quantity_units = converted_quantity_units) %>%
   bind_rows(dropbox_impacts) %>%
   rename(impact_category = impactCategory) 

write.csv(decon, "intermediary/deconData.csv")
```

### Data wrangling and objects for transport calculations

```{r material weight}
project_material_weight_sum <- salvage_materials_weight %>%
   group_by(project, deconMaterialName) %>%
   summarise(total_salvage_quantity = sum(converted_quantity)) %>%
   add_column(quantity_units = "kg")

write.csv(project_material_weight_sum, "intermediary/project_material_weight_sum.csv")
```


## Demolition equivalent on salvaged materials

In order to obtain net benefits, the deconstruction scenario must be compared to the approximately equivalent demolition scenario. For materials that were salvaged, we will need to reassign the reuse dispositions accordingly and re-calculate the energy and carbon impacts.

```{r demo scenario data}
# assign NonSalvage material dispositions 
demoEOL <- EOL[38:66, c(3:4,7:9)] %>%
   gather(`Percent Recycled`, `Percent Incinerated`, `Percent Landfilled`,  key = "disposition_assignment", value = "percentage") %>%
   mutate(disposition = case_when(
      .$disposition_assignment == "Percent Recycled" ~ "recyclingGeneric",
      .$disposition_assignment == "Percent Incinerated" ~ "incineration",
      .$disposition_assignment == "Percent Landfilled" ~ "landfill"
   ))

demo_scenario <- salvage_materials_weight %>%
   left_join(demoEOL, by = "deconMaterialName") %>%
   inner_join(demo_impacts, by = c("deconMaterialName", "disposition")) %>%
   mutate(percent_EOL_quantity = converted_quantity * percentage) %>%
   mutate(material_impacts = percent_EOL_quantity * impactValue) %>%
   mutate(material_impact_units = impactUnit) %>%
   rename(quantity = percent_EOL_quantity) %>%
   rename(quantity_units = converted_quantity_units) %>%
   select("house_age", "house_size", "project", "contractor", "deconMaterialName", "quantity", "quantity_units", "disposition", "impactCategory", "material_impacts", "material_impact_units", "SimpleEOLname")

```

Cleaning up the `demo_scenario` data frame yields the `demo` data that can be used to differentiate net benefits from the `decon` data.

```{r demo}
demo <- demo_scenario %>%
   bind_rows(dropbox_impacts) %>%
   rename(impact_category = impactCategory) %>%
   filter(quantity != 0)

write.csv(demo, "intermediary/demoData.csv")
```
   
   
# Data Summary and Further Use

The two scenario data frames are saved as separate files and each contain the following variables:
`r names(demo)`

To accurately represent net benefits of deconstruction over demolition, these figures must also account for transportation impacts. The data prepared in this notebook will be used in `03_Transport.Rmd` to determine the material based transport impacts. Once the transport related impacts are calculated, the `decon` and `demo ` data frames will be combined with the transport impacts in `04_decon_data_analysis2018` and the net benefits calculated within a final report.




