Replicant

h1. RestoreApplicationInternalData

{{toc}}

h2. /!\ Warning: Draft

This article is in draft form and is being written:

* Everybody is welcome to contribute

* Some things might not be accurate yet, so beware before using the information contained in it.

h2. Rationale

In some case, it is useful to be able to restore internal applications data:

For instance you might need to move the data of an application from a device to another if you want to switch device.

Another use case is if /data/system/packages.xml or /data/system/appops.xml get corrupted, applications can loose access to their data. This can make the launcher and other applications crash. 

So while it is possible to recover from that by wiping the data partition in the recovery, sometimes it's very impractical to do that because you might have important data like silence encryption keys and established sessions that you don't want to loose.

h2. Goals

This howto will explain how to move silence data from a device to another.

For instance you could want to move from a Galaxy SIII (GT-I9300) to a Galaxy SII (GT-I9100) or vice versa, and you might not want to recreate keys, sessions, etc when moving device.

This could also be adapted to restore application data from a block device level backup of the internal eMMC or the internal data partition.

h2. Silence

Silence has been chosen as an example for this tutorial because:

* It's an application commonly used

* Loosing its data (key, sessions) can be painful

Silence stores its data in the internal application storage. As far as I know it's not supposed to store any data on the microSD or user storage beside potential backups.

It might be interesting to make additional tutorial for other cases. For instance for:

* Applications that also require data to be on the microSD or user storage.

* System applications that have their data in a database.

h2. Requirements 

This howto assumes that the data partition is unencrypted. If you know how to open encrypted data partition in the recovery, or in a GNU/Linux distribution, it would be great to either modify this tutorial to add information on how to do it, or contact us on the mailing list or through the bugreporting system about it.

h2. TODO

* Explain why we need the uid/gid

* Explain why the ls -ld gives the application uid/gid

* Point to how to get root

* Explain how to handle a corrupted /data/system/packages.xml and /data/system/appops.xml

* Explain how to mount a full backup, and why not to restore full backup completely

* Explain how and why create a tarball of the application data

Once we have a tarball backup of the application data we need to reboot in the recovery to avoid any writes to the data filesystem.

h2. Backuping Silence's data from the old device

To correctly backup the application data, you need to make sure that the application isn't writing to its data while you are doing the backup and that the data is consistent.

To make sure of that the easiest way to do that is to go in the recovery: this way the application will not be started automatically and the data is consistent. You will also need a root shell there, so make sure to setup that beforehand.

If everything is setup correctly, running @adb shell@ from your computer should result in a shell that now looks like this:

<pre>

root@m0:/ #

</pre>

Once that you have a shell in the recovery, you need to mount the data partition in /data from within that shell.

For the Galaxy SIII (GT-I9300), this can be done from your computer with this command:

<pre>

mount /dev/block/platform/*/by-name/USERDATA /data

</pre>

We will then create an archive of the silence data. 

This makes things much easier because with an archive:

* We can easily store and move that data.

* The archive preserves all its permissions, and it's best to keep them as applications might not work correctly otherwise. For instance in GNU/Linux, the 'sshd' program will refuse to work if some of its configuration files have the wrong permissions.

The silence data is in @/data/data/org.smssecure.smssecure/@. To make sure that the archive is created correctly and that once decompressed it will create the @org.smssecure.smssecure@ directory (with all the silence data in it) in the current directory, we need to go to @/data/data@ first.

To do that, we can go in /data/data with the following command:

<pre>

cd /data/data

</pre>

We can then create the archive in the recovery with that command:

<pre>

root@m0:/data/data # tar cvpf /org.smssecure.smssecure.tar org.smssecure.smssecure/

</pre>

This will create the @/org.smssecure.smssecure.tar@ file which is our archive.

We will then exit the recovery shell as we will then need to type commands on your computer.

You can exit the recovery shell with the following command:

<pre>

root@m0:/ # exit

</pre>

We're now back on your computer.

As for now, we've created the archive but it's still in the recovery, so we need to copy it to your computer. 

This can be done with the following command:

<pre>

$ adb pull /org.smssecure.smssecure.tar ./

</pre>

It will create the @org.smssecure.smssecure.tar@ file in the current directory.

We will then need to verify that the archive was created correctly. 

More precisely we need to check that it will be extracted in the @org.smssecure.smssecure/@ directory, in the current directory, otherwise it could mess up the data partition when it's being uncompressed.

To do that we can list its content with the following command:

<pre>

$ tar tvf org.smssecure.smssecure.tar

</pre>

It should output something that will look more or less like that:

<pre>

drwxr-x--x u0_a61/u0_a61     0 2020-10-28 18:58 org.smssecure.smssecure/

lrwxrwxrwx root/root         0 2020-10-28 18:58 org.smssecure.smssecure/lib -> /data/app/org.smssecure.smssecure-1/lib/arm

[...]

</pre>

What is important to check is that the files and directories starts with @org/smssecure.smssecure/@ or @./org.smssecure.smssecure/@. If not something went wrong during the creation of the backup (for instance you might have forgetten to go in the @/data/data/@ recovery, or this tutorial could contain mistakes or be out of date).

When all that is done, you can power off the device by going going in @Advanced@ in the recovery menu and then selecting @>Power off@. If you want to reboot to Replicant instead you could select @Reboot system now@ in the recovery menu instead.

h2. Restoring Silence's data to the new device

Once the backup is done you need to mount the data partition. 

For the Galaxy SIII (GT-I9300), this can be done from your computer with this command:

<pre>

$ adb shell "mount /dev/block/platform/dw_mmc/by-name/USERDATA /data"

</pre>

Then we need to get a root shell inside the recovery. This can be done with the @adb shell@ command:

<pre>

$ adb shell

</pre>

Then we assume that you are in /data/data to simplify this tutorial.

You will need to remember adjust all other commands if you are not in this directory.

To go in /data/data, you can use the following command:

<pre>

root@m0:/ # cd /data/data/                                                     

root@m0:/data/data # 

</pre>

As applications are sandboxed, and that as part of that sandboxing, they have their own usersname, we need to retrieve this username.

To do that we can just use @ls -ld@ on the directory holding the application internal data.

The directory has the internal name of the application.

Here are some well known name correspondances:

| Internal name           | Application                                 |

| org.smssecure.smssecure | Silence                                     |

| com.android.dialer      | Dialer (Android's stock dialer application) |

| fil.libre.repwifiapp    | RepWiFi                                     |

For pakcages comming from f-droid, the f-droid website can find the correspondance.

For instance the "Silence page":https://f-droid.org/en/packages/org.smssecure.smssecure/ has @org.smssecure.smssecure@ in its URL and inside the page.

So with @ls -ld@ we can find the application username in this way:

<pre>

root@m0:/data/data # ls -ld org.smssecure.smssecure

__bionic_open_tzdata: couldn't find any tzdata when looking for localtime!

__bionic_open_tzdata: couldn't find any tzdata when looking for GMT!

__bionic_open_tzdata: couldn't find any tzdata when looking for posixrules!

drwxr-x--x 2 u0_a61 u0_a61 4096 2012-01-01 00:01 org.smssecure.smssecure

</pre>

Here the users and groups are @u0_a61@.

We will then need this information later on to restore the silence data from the other device: If we restore that data as-is it will most likely have wrong permissions: when the the silence application was installed on the older device, it was assigned an username. As this username depends on the number of applications that were installed before it, we cannot expect it to always be the same between the two devices.

It's also best to move or delete the data we don't want:

<pre>

root@m0:/data/data # mv org.smssecure.smssecure org.smssecure.smssecure.delme

</pre>

Moving it has several advantages:

* We can still verify the username later on to see if it matches with the backup we restored.

* We can interrupt this tutorial more easily if something goes wrong.

Here we need to verify that the archive will extract its files in the @org.smssecure.smssecure@ directory and not in the current directory which is @/data/data@:

<pre>

root@m0:/data/data # tar tf /org.smssecure.smssecure.tar

./org.smssecure.smssecure/

./org.smssecure.smssecure/lib -> /data/app/org.smssecure.smssecure-1/lib/arm

[...]

</pre>

Here we see that everything starts with @./org.smssecure.smssecure/@ (or @org.smssecure.smssecure/@) so it's good.

TODO: move this part earlier

If we had something like that instead:

<pre>

root@m0:/data/data # tar tf /org.smssecure.smssecure.tar

./lib -> /data/app/org.smssecure.smssecure-1/lib/arm

[...]

</pre>

Then it's best to recreate the archive.

If you need more time you could also move back org.smssecure.smsecure.delme to org.smssecure.smssecure if needed.

We can then proceed to extract the application data (with the username from the old device):

<pre>

root@m0:/data/data # tar xpf /org.smssecure.smssecure.tar --numeric-owner

</pre>

Here we can see that the username differs from the one we need:

<pre>

root@m0:/data/data # ls -ld org.smssecure.smssecure 

__bionic_open_tzdata: couldn't find any tzdata when looking for localtime!

__bionic_open_tzdata: couldn't find any tzdata when looking for GMT!

__bionic_open_tzdata: couldn't find any tzdata when looking for posixrules!

drwxr-x--x 9 u0_a63 u0_a63 4096 2012-01-01 00:21 org.smssecure.smssecure

</pre>

We have @u0_a63@ instead of @u0_a61@.

So we need to fix it. This can be done with the @chown@ command, like that:

<pre>

root@m0:/data/data # chown u0_a61:u0_a61 -R org.smssecure.smssecure            

root@m0:/data/data # 

</pre>

At this point, we don't need the @org.smssecure.smssecure.delme@ directory anymore, and it's best to remove it not to create any issues later on.

This can be done with the following command:

<pre>

root@m0:/data/data # rm -rf org.smssecure.smssecure.delme

root@m0:/data/data # 

</pre>

In addition to the standard unix permissions, Android also uses selinux, so we also need to fixup the selinux permissions.

The restorecon command can be used for that. Here's its help: 

<pre>

root@m0:/data/data # restorecon                                                

usage: restorecon [-D] [-F] [-R] [-n] [-v] FILE...

Restores the default security contexts for the given files.

-D	apply to /data/data too

-F	force reset

-R	recurse into directories

-n	don't make any changes; useful with -v to see what would change

-v	verbose: show any changes

restorecon: Needs 1 argument

</pre>

So to use it to fixup the selinux permissions, we can use the following command:

<pre>

root@m0:/data/data # restorecon -D -F -R -v /data/

</pre>

The order of the arguments (-D, -F, etc) seem to be important here as the wrong order might result in nothing being done.

Without the @-v@ argument and with the wrong order of argument, it might make you think that it did its job while it did nothing.

It will then print something that looks like that:

<pre>

SELinux: Loaded file_contexts contexts from /file_contexts.

[...]

SELinux:  Relabeling /data/data/org.smssecure.smssecure from u:object_r:system_data_file:s0 to u:object_r:app_data_file:s0:c512,c768.

SELinux:  Relabeling /data/data/org.smssecure.smssecure/lib from u:object_r:system_data_file:s0 to u:object_r:app_data_file:s0:c512,c768.

[...]

</pre>

The premissions fixing is now done.

So we can then umount the data partition and reboot.

To do that first we need to go outside of data, else the mount will fail:

<pre>

root@m0:/data/data # cd /For instance you could want to move from a Galaxy SIII (GT-I9300) to a Galaxy SII (GT-I9100) or vice versa, and you might not want to recreate keys, sessions, etc when moving device.

root@m0:/ # 

</pre>

Then we can simply unmount it with this command:

<pre>

root@m0:/ # umount  /data/                                                     

root@m0:/ # 

</pre>

Then it's a good practice to make sure that everything is written to the data partition before rebooting.

We can do that with the @sync@ command:

<pre>

root@m0:/ # sync

</pre>

And we can finally reboot:

<pre>

root@m0:/ # reboot

</pre>

After rebooting, silence still refused to start.

So I looked at the logs from my laptop with this command:

<pre>

$ adb logcat -b main

</pre>

I waited until no more new logs were printed.

I then press enter multiple times to create a separation with many new lines to be able to get back to the begining of the new logs easily.

Then I launched silence and started pressing enter multiple times again to mark the end of the silence related logs.

I then had that in these new logs:

<pre>For instance you could want to move from a Galaxy SIII (GT-I9300) to a Galaxy SII (GT-I9100) or vice versa, and you might not want to recreate keys, sessions, etc when moving device.

01-01 01:27:48.260  4126  4126 D AndroidRuntime: Shutting down VM

01-01 01:27:48.265  4126  4126 E AndroidRuntime: FATAL EXCEPTION: main

01-01 01:27:48.265  4126  4126 E AndroidRuntime: Process: org.smssecure.smssecure, PID: 4126

01-01 01:27:48.265  4126  4126 E AndroidRuntime: Theme: themes:{}

01-01 01:27:48.265  4126  4126 E AndroidRuntime: java.lang.RuntimeException: Unable to create application org.smssecure.smssecure.ApplicationContext: java.lang.SecurityException: getActiveSubscriptionInfoList: Neither user 10061 nor current process has android.permission.READ_PHONE_STATE.

01-01 01:27:48.265  4126  4126 E AndroidRuntime: 	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4754)

[...]

</pre>

So to fix it I went in @Settings->Apps->Silence->Permissions@ and gave it all the permissions it needed.

I had this issue because I didn't even launch silence after installing it, so it cound't ask me for the permissions it needed.

And the silence of the former device probably wrote in its data that it already asked the permissions.