背景

日常某应用长年累月运行时，需要知道其crash数据是否有改善，而由于crash平台（bugly、flurry、sentry）是滚动时间存储的，所以为了留痕，需要设计一个平台抓取以上数据，做数据分析、汇总，应该具备：

1. 获取全部异常的数据，按天、小时分表存储，便于回溯；
2. 应该提供按月维度的展示列表，应该支持android、iOS、Harmony等平台；
3. 应该支持celery自动抓取；
4. 支持异常情况修正数据；

设计

Account
|
App
|   |
AppVersion  CrashForProductionVersion
|   |   |
CrashData  CrashDataAmendment  CrashDataHourly

核心模型设计：

python3
class Account(models.Model):

    @cached_property
    def crawler(self):
        pass
        
    def login(self):
        pass
        
    def ensure_login(self):
        pass
        
    def sync_apps(self):
        pass
        
class CrashData(models.Model):
    date_report = models.DateField(
        verbose_name='上报日期', db_index=True)
    access_num = models.IntegerField(
        verbose_name='启动次数', default=-1)
    access_user = models.IntegerField(
        verbose_name='联网设备数', default=-1)
    crash_num = models.IntegerField(
        verbose_name='崩溃次数', default=-1)
    crash_user = models.IntegerField(
        verbose_name='影响设备数', default=-1)
    crash_type = models.CharField(
        verbose_name='崩溃类型', max_length=50,
        choices=CRASH_TYPE_CHOICES, default=CRASH_TYPE_CRASH)

抓取模型设计：

python3
class CrawlerBase:
    SUPPORTED_CRASH_TYPES = ()

    def __init__(self, context_name, ctx: UserContext = None, domain=''):
        self.context_name = context_name
        self.domain = domain
        self.ctx = ctx or UserContext(context_name)

    def login(self, username='', password='', *args, **kwargs) -> UserContext:
        raise NotImplementedError()

    def login_with_curl_script(self, script) -> UserContext:
        raise NotImplementedError()

    def check_login(self) -> bool:
        raise NotImplementedError()

    def ensure_login(self) -> bool:
        if self.check_login():
            return True
        self.login()
        return self.check_login()

    def fetch_app_list(self) -> [dict]:
        raise NotImplementedError()

    def fetch_version_list(self, app) -> [dict]:
        raise NotImplementedError()

    def fetch_daily_crash(self, app, version, date_begin, date_end, crash_type):
        raise NotImplementedError()

    def fetch_hourly_crash(
            self, app: dict, version: dict, date_report: date = None, crash_type='crash'):
        raise NotImplementedError()

    def get_scrap_date_intervals(self, last_date=None):
        """
        Returns the full range of dates for data extraction (from newest to oldest), implemented based on platform characteristics (by year or by month).
        :param last_date: The last date of extraction, from which the time intervals are generated backward.
        """

        raise NotImplementedError()